mdbook-tsitter

Tree-sitter syntax highlighting for mdBook — for any language you can point a grammar at.

mdbook-tsitter is an mdBook preprocessor that parses fenced code blocks with tree-sitter and renders the resulting captures as themeable HTML. It is grammar-agnostic: supply a compiled parser and the grammar’s highlight queries, and the same preprocessor can handle anything from Rust and Lua to a language of your own.

Features

Grammar-agnostic. Use any tree-sitter grammar that can be built as a shared library.
Structural highlighting. Definitions, calls, types, parameters, properties, macros, and other syntax can receive distinct styles whenever the grammar’s queries distinguish them.
Embedded languages. Injection queries can highlight fenced code inside Markdown, languages embedded in strings, and other nested syntax.
Themeable capture classes. Captures become predictable ts-… CSS classes, with a bundled stylesheet that follows mdBook’s light, rust, coal, navy, and ayu themes.
Language aliases and locals queries. Match several fence names to one grammar and opt into scope-aware query behavior where the grammar supports it.
Selective processing. Leave individual blocks to mdBook when you need its built-in Rust Playground behavior or other native code-block features.
Build-time rendering. Highlighted HTML is generated with the book; the reader does not need a tree-sitter runtime.

The example book includes a custom Macaulay2 grammar and substantial side-by-side examples in Rust, Python, TypeScript, JavaScript, Go, Java, C, C++, Bash, PHP, Lua, and Haskell. Its complete source lives in examples/languages.

Installation

Install the preprocessor from crates.io:

cargo install mdbook-tsitter

The mdbook-tsitter binary must be available on PATH when mdbook build runs. Each configured language also needs:

A compiled tree-sitter parser (.so, .dylib, or .dll).
Its queries/highlights.scm file.
Optionally, injections.scm and locals.scm.

See Getting parsers and queries for common ways to obtain them.

Quick start

Generate the default stylesheet in your book:

mkdir -p theme
mdbook-tsitter css > theme/treesitter.css

Place a parser and its highlight query in your project, then configure book.toml:

[preprocessor.tsitter]

[preprocessor.tsitter.languages.rust]
library = "parsers/rust.so"
highlights = "queries/rust/highlights.scm"

[output.html]
additional-css = ["theme/treesitter.css"]

Paths are relative to the book root. After that, ordinary fenced blocks for the configured language are highlighted automatically:

```rust
fn main() {
    println!("highlighted at build time");
}
```

Build the book as usual:

mdbook build

Languages that are not configured, blocks without a language tag, and blocks that opt out are left untouched for mdBook to handle.

Configuring languages

Everything lives under [preprocessor.tsitter].

Preprocessor options

key	default	meaning
`inject`	`true`	Highlight embedded languages using configured injection queries. Only configured languages are used. Set this to `false` to disable injections globally.

Language options

Add one [preprocessor.tsitter.languages.<name>] table per grammar:

key	required	meaning
`library`	yes	Path to the compiled parser shared library.
`highlights`	yes	Path to the grammar’s highlights query.
`symbol`	no	Parser constructor symbol; defaults to `tree_sitter_<name>` with `-` changed to `_`.
`injections`	no	Path to an injections query for embedded languages.
`locals`	no	Path to a locals query for scope-aware highlighting.
`aliases`	no	Additional code-fence names handled by this grammar; defaults to the table key.

A fuller Markdown configuration might look like this:

[preprocessor.tsitter.languages.markdown]
library = "parsers/markdown.so"
highlights = "queries/markdown/highlights.scm"
injections = "queries/markdown/injections.scm"
aliases = ["md", "markdown"]

Injection queries can select another configured grammar using standard @injection.language and @injection.content captures. All grammars share the same capture-class table during a build, so embedded and host languages use the same theme consistently.

Leaving a block to mdBook

Processing a block replaces its Markdown with highlighted HTML. Add the notreesitter annotation when a block should retain mdBook’s own handling, including Rust Playground buttons, hidden lines, and ignore or no_run annotations:

```rust,notreesitter
# fn main() {
let runnable = "mdBook keeps control of this block";
# }
```

Getting parsers and queries

Most grammars live in a tree-sitter-<language> repository. With the tree-sitter CLI, a typical parser build looks like:

git clone https://github.com/<owner>/tree-sitter-nix
cd tree-sitter-nix
tree-sitter build --output libtree-sitter-nix.so

Use .dylib on macOS or .dll on Windows, and point library at the exact output path. The extension is never assumed.

The highlight query is usually the grammar’s queries/highlights.scm. An existing nvim-treesitter installation is another convenient source of compiled parsers and queries.

The setup script in examples/languages/setup.sh shows one way to stage several parsers and their queries for a complete book.

Theming

The generated stylesheet covers standard tree-sitter and nvim-treesitter capture names and adapts to mdBook’s built-in colour schemes. Edit it directly or add a later stylesheet with your overrides.

Each capture gets the ts- prefix, dots become hyphens, and every prefix is emitted so broad rules can cascade into more specific ones:

capture	generated classes
`keyword`	`ts-keyword`
`keyword.operator`	`ts-keyword ts-keyword-operator`
`string.regexp`	`ts-string ts-string-regexp`

For example:

.ts-comment {
  font-style: italic;
}

code.language-rust .ts-function-macro {
  color: var(--ts-purple);
}

There is no fixed capture list in the preprocessor. Capture names come directly from each grammar’s queries; names beginning with _ are treated as internal and left unstyled.

How it works

mdBook passes every chapter to the preprocessor as Markdown. mdbook-tsitter uses the same Markdown parser as mdBook (pulldown-cmark), locates top-level fenced code blocks for configured languages, and sends their source through tree-sitter-highlight.

The highlighted events become semantic CSS classes and are spliced back into the chapter as ready-made HTML:

<pre class="treesitter"><code class="no-highlight language-rust">…spans…</code></pre>

The original language class remains available for per-language styles, while no-highlight marks the HTML as already processed.

Development

Run the Rust test suite:

cargo test

Build the multi-language example book:

./examples/languages/setup.sh
mdbook build examples/languages

The parsers and queries used by the deployed example are committed so the hosted build is reproducible. The setup script is useful when refreshing those assets from local grammar installations.

License

Licensed under either of MIT or Apache-2.0 at your option.

Rust

Traits, generic bounds, Arc delegation, concurrent maps, locks, iterators, closures, and let-else.

  mdBook
  tree-sitter

#![allow(unused)]
fn main() {
use dashmap::DashMap;
use lsp_types::{Location, Range, Url};
use std::{path::PathBuf, sync::{Arc, RwLock}};

#[derive(Debug, Clone)]
struct DefSite {
    uri: Url,
    range: Range,
    kind: TokenKind,
}

trait DefLookup {
    fn find(&self, name: &str, skip: &Url)
        -> Vec<Location>;
    fn has(&self, name: &str) -> bool;
    fn kind(&self, name: &str, skip: &Url)
        -> Option<TokenKind>;
}

impl<T: DefLookup + ?Sized> DefLookup for Arc<T> {
    fn find(
        &self,
        name: &str,
        skip: &Url,
    ) -> Vec<Location> {
        self.as_ref().find(name, skip)
    }

    fn has(&self, name: &str) -> bool {
        self.as_ref().has(name)
    }

    fn kind(
        &self,
        name: &str,
        skip: &Url,
    ) -> Option<TokenKind> {
        self.as_ref().kind(name, skip)
    }
}

/// A symbol index kept in sync with editor changes.
#[derive(Debug, Default)]
struct SymbolIndex {
    defs: DashMap<Symbol, Vec<DefSite>>,
    files: DashMap<Url, Vec<Symbol>>,
    roots: RwLock<Vec<PathBuf>>,
}

impl SymbolIndex {
    fn set_roots(&self, roots: Vec<PathBuf>) {
        *self.roots.write().expect("roots lock") = roots;
    }

    fn roots(&self) -> Vec<PathBuf> {
        self.roots.read().expect("roots lock").clone()
    }

    fn index<K>(&self, uri: &Url, text: &str, p: &K)
    where
        K: TypeProvider + ?Sized,
        for<'a> K::View<'a>: TokenKnowledge,
    {
        self.remove(uri);
        let defs = top_level_defs(text, p);
        if defs.is_empty() {
            return;
        }

        let mut names = Vec::with_capacity(defs.len());
        for (name, range, kind) in defs {
            let name = Symbol::new(&name);
            self.defs
                .entry(name.clone())
                .or_default()
                .push(DefSite {
                    uri: uri.clone(),
                    range,
                    kind,
                });
            names.push(name);
        }
        self.files.insert(uri.clone(), names);
    }

    fn remove(&self, uri: &Url) {
        let Some((_, names)) = self.files.remove(uri)
        else {
            return;
        };

        for name in names {
            let empty = self.defs
                .get_mut(&name)
                .is_some_and(|mut sites| {
                    sites.retain(|site| &site.uri != uri);
                    sites.is_empty()
                });
            if empty {
                self.defs.remove(&name);
            }
        }
    }
}

impl DefLookup for SymbolIndex {
    fn find(
        &self,
        name: &str,
        skip: &Url,
    ) -> Vec<Location> {
        self.defs.get(name)
            .into_iter()
            .flat_map(|sites| sites.iter())
            .filter(|site| &site.uri != skip)
            .map(|site| Location {
                uri: site.uri.clone(),
                range: site.range,
            })
            .collect()
    }

    fn has(&self, name: &str) -> bool {
        self.defs.contains_key(name)
    }

    fn kind(
        &self,
        name: &str,
        skip: &Url,
    ) -> Option<TokenKind> {
        self.defs.get(name)?
            .iter()
            .find(|site| &site.uri != skip)
            .map(|site| site.kind)
    }
}
}

use dashmap::DashMap;
use lsp_types::{Location, Range, Url};
use std::{path::PathBuf, sync::{Arc, RwLock}};

#[derive(Debug, Clone)]
struct DefSite {
    uri: Url,
    range: Range,
    kind: TokenKind,
}

trait DefLookup {
    fn find(&self, name: &str, skip: &Url)
        -> Vec<Location>;
    fn has(&self, name: &str) -> bool;
    fn kind(&self, name: &str, skip: &Url)
        -> Option<TokenKind>;
}

impl<T: DefLookup + ?Sized> DefLookup for Arc<T> {
    fn find(
        &self,
        name: &str,
        skip: &Url,
    ) -> Vec<Location> {
        self.as_ref().find(name, skip)
    }

    fn has(&self, name: &str) -> bool {
        self.as_ref().has(name)
    }

    fn kind(
        &self,
        name: &str,
        skip: &Url,
    ) -> Option<TokenKind> {
        self.as_ref().kind(name, skip)
    }
}

/// A symbol index kept in sync with editor changes.
#[derive(Debug, Default)]
struct SymbolIndex {
    defs: DashMap<Symbol, Vec<DefSite>>,
    files: DashMap<Url, Vec<Symbol>>,
    roots: RwLock<Vec<PathBuf>>,
}

impl SymbolIndex {
    fn set_roots(&self, roots: Vec<PathBuf>) {
        *self.roots.write().expect("roots lock") = roots;
    }

    fn roots(&self) -> Vec<PathBuf> {
        self.roots.read().expect("roots lock").clone()
    }

    fn index<K>(&self, uri: &Url, text: &str, p: &K)
    where
        K: TypeProvider + ?Sized,
        for<'a> K::View<'a>: TokenKnowledge,
    {
        self.remove(uri);
        let defs = top_level_defs(text, p);
        if defs.is_empty() {
            return;
        }

        let mut names = Vec::with_capacity(defs.len());
        for (name, range, kind) in defs {
            let name = Symbol::new(&name);
            self.defs
                .entry(name.clone())
                .or_default()
                .push(DefSite {
                    uri: uri.clone(),
                    range,
                    kind,
                });
            names.push(name);
        }
        self.files.insert(uri.clone(), names);
    }

    fn remove(&self, uri: &Url) {
        let Some((_, names)) = self.files.remove(uri)
        else {
            return;
        };

        for name in names {
            let empty = self.defs
                .get_mut(&name)
                .is_some_and(|mut sites| {
                    sites.retain(|site| &site.uri != uri);
                    sites.is_empty()
                });
            if empty {
                self.defs.remove(&name);
            }
        }
    }
}

impl DefLookup for SymbolIndex {
    fn find(
        &self,
        name: &str,
        skip: &Url,
    ) -> Vec<Location> {
        self.defs.get(name)
            .into_iter()
            .flat_map(|sites| sites.iter())
            .filter(|site| &site.uri != skip)
            .map(|site| Location {
                uri: site.uri.clone(),
                range: site.range,
            })
            .collect()
    }

    fn has(&self, name: &str) -> bool {
        self.defs.contains_key(name)
    }

    fn kind(
        &self,
        name: &str,
        skip: &Url,
    ) -> Option<TokenKind> {
        self.defs.get(name)?
            .iter()
            .find(|site| &site.uri != skip)
            .map(|site| site.kind)
    }
}

C++

Concepts, ranges, views, variants, smart pointers, lambdas, and move semantics.

  mdBook
  tree-sitter

#include <algorithm>
#include <concepts>
#include <iostream>
#include <memory>
#include <ranges>
#include <string>
#include <variant>
#include <vector>

struct Text {
    std::string value;
};

struct Number {
    double value;
};

using Cell = std::variant<Text, Number>;

template<typename T>
concept Printable = requires(
    std::ostream& out,
    const T& value
) {
    { out << value } -> std::same_as<std::ostream&>;
};

class Table {
public:
    void add(Cell cell) {
        cells_.push_back(std::move(cell));
    }

    auto numbers() const {
        return cells_
            | std::views::filter([](const Cell& cell) {
                return std::holds_alternative<Number>(
                    cell
                );
            })
            | std::views::transform([](const Cell& cell) {
                return std::get<Number>(cell).value;
            });
    }

private:
    std::vector<Cell> cells_;
};

template<std::ranges::input_range R>
requires std::convertible_to<
    std::ranges::range_value_t<R>,
    double
>
double average(R&& values) {
    double sum = 0.0;
    std::size_t count = 0;
    for (double value : values) {
        sum += value;
        ++count;
    }
    return count ? sum / count : 0.0;
}

int main() {
    auto table = std::make_unique<Table>();
    table->add(Text{"latency"});
    table->add(Number{18.4});
    table->add(Number{21.2});

    std::cout << average(table->numbers()) << '\n';
}


#include <algorithm>
#include <concepts>
#include <iostream>
#include <memory>
#include <ranges>
#include <string>
#include <variant>
#include <vector>

struct Text {
    std::string value;
};

struct Number {
    double value;
};

using Cell = std::variant<Text, Number>;

template<typename T>
concept Printable = requires(
    std::ostream& out,
    const T& value
) {
    { out << value } -> std::same_as<std::ostream&>;
};

class Table {
public:
    void add(Cell cell) {
        cells_.push_back(std::move(cell));
    }

    auto numbers() const {
        return cells_
            | std::views::filter([](const Cell& cell) {
                return std::holds_alternative<Number>(
                    cell
                );
            })
            | std::views::transform([](const Cell& cell) {
                return std::get<Number>(cell).value;
            });
    }

private:
    std::vector<Cell> cells_;
};

template<std::ranges::input_range R>
requires std::convertible_to<
    std::ranges::range_value_t<R>,
    double
>
double average(R&& values) {
    double sum = 0.0;
    std::size_t count = 0;
    for (double value : values) {
        sum += value;
        ++count;
    }
    return count ? sum / count : 0.0;
}

int main() {
    auto table = std::make_unique<Table>();
    table->add(Text{"latency"});
    table->add(Number{18.4});
    table->add(Number{21.2});

    std::cout << average(table->numbers()) << '\n';
}

Macaulay2

Self-initializing types, methods, typed dispatch, hash and cache tables, regular expressions, recursion, loops, control flow, and tree transformations.

Metavar = new SelfInitializingType of TokenTree

metavarNode = method()
metavarNode String := Metavar => name -> Metavar(name, {}, null, null)
metavarNode (String, String) := Metavar => (name, kind) -> Metavar(name, {}, null, kind)

isMetavar = t -> instance(t, Metavar)
metavarName = t -> leftOf t
metavarKind = t -> delimiterOf t

Repetition = new SelfInitializingType of TokenTree

repetitionNode = (quantifier, sep, unit) -> Repetition(quantifier, unit, null, sep)
isRepetition = t -> instance(t, Repetition)
repQuantifier = t -> leftOf t
repSeparator = t -> delimiterOf t
repUnit = t -> contentOf t

Alternation = new SelfInitializingType of TokenTree

alternationNode = branches -> Alternation(null, branches, null, null)
isAlternation = t -> instance(t, Alternation)
altBranches = t -> contentOf t

isSeqNode = t -> not isRepetition t and
    (delimiterOf t === "," or delimiterOf t === ";" or delimiterOf t === statementSeparator)

nodeKind = t -> (
    if isComment t then "Comment"
    else if isMacroInvocation t then "MacroInvocation"
    else if isMetavar t then "Metavar"
    else if isRepetition t then "Repetition"
    else if isAlternation t then "Alternation"
    else if isLeaf t then (
        s := leftOf t;
        if s === null then "Node"
        else if s#0 == "\"" then "String"
        else if match("^[0-9]", s) then "Number"
        else if match("^[A-Za-z]", s) then (if m2Keywords#?s then "Keyword" else "Identifier")
        else "Operator")
    else (
        d := delimiterOf t;
        if d === spaceOperator then "Apply"
        else if d === statementSeparator then "Statements"
        else if d === whitespaceDelimiter then (
            cs := contentOf t;
            if #cs == 0 then "Clause" else capitalize leftOf cs#0)
        else if d === "," or d === ";" then "Sequence"
        else if d === "->" then "Arrow"
        else if instance(d, String) then "Infix"
        else if leftOf t =!= null and rightOf t =!= null then "Bracket"
        else if leftOf t =!= null then "Prefix"
        else if rightOf t =!= null then "Postfix"
        else "Node"))

nodeKindNames = set {"Comment", "MacroInvocation", "Metavar", "Repetition", "Alternation", "String",
    "Number", "Keyword", "Identifier", "Operator", "Apply", "Sequence", "Arrow",
    "Infix", "Bracket", "Prefix", "Postfix", "If", "While", "For", "Try", "New",
    "Statements", "Clause", "Node"}

metavarPlaceholderPrefix = "MetavarHolePlaceholder"
metavarKindPrefix = "MetavarKind"
toPlaceholders = src -> (
    typed := replace(///(?<![A-Za-z0-9'])'([A-Za-z][A-Za-z0-9]*):([A-Za-z][A-Za-z0-9]*)///,
        concatenate(metavarKindPrefix, "$2(", metavarPlaceholderPrefix, "$1)"), src);
    replace(///(?<![A-Za-z0-9'])'([A-Za-z][A-Za-z0-9]*)///, metavarPlaceholderPrefix | "$1", typed))

repCallNames = new HashTable from {"+" => "RepPlus", "*" => "RepStar", "|" => "Alt"}
isIdentChar = c -> match("[A-Za-z0-9']", c)
scanReps = src -> (
    n := #src;
    at := i -> if i >= 0 and i < n then substring(i, 1, src) else "";
    stack := {};
    spans := {};
    for i to n - 1 do (
        c := at i;
        if c == "{" then (
            isFormOpen := i >= 1 and at(i - 1) == "'" and (i < 2 or not isIdentChar at(i - 2));
            stack = append(stack, (i, isFormOpen)))
        else if c == "}" then (
            if #stack == 0 then error "scanReps: unbalanced }";
            top := last stack;
            stack = drop(stack, -1);
            if top#1 then (
                form := if at(i + 1) == "+" or at(i + 1) == "*" then at(i + 1) else "|";
                spans = append(spans, (top#0 - 1, i, form)))));
    if #spans == 0 then return src;
    opens := hashTable apply(spans, s -> (s#0, repCallNames#(s#2) | "("));
    closes := hashTable apply(spans, s -> (s#1, if s#2 === "|" then 1 else 2));
    out := "";
    j := 0;
    while j < n do (
        if opens#?j then (out |= opens#j;
            j += 2)
        else if closes#?j then (out |= ")";
            j += closes#j)
        else (out |= at j;
            j += 1));
    out)

quantifierOf = t -> (
    if delimiterOf t === spaceOperator and #contentOf t == 2 and isLeaf (contentOf t)#0
    then (n := leftOf (contentOf t)#0;
        if n === "RepPlus" then "+" else if n === "RepStar" then "*"))

isNullElement = t -> isLeaf t and leftOf t === "null"
unitOf = t -> (
    inner := (contentOf (contentOf t)#1)#0;
    sep := if isSeqNode inner then delimiterOf inner else ",";
    elems := if isSeqNode inner then contentOf inner else {inner};
    while #elems > 0 and isNullElement last elems do elems = drop(elems, -1);
    (sep, elems))

altCallName = "Alt"
altInnerOf = t -> (
    if delimiterOf t === spaceOperator and #contentOf t == 2 and isLeaf (contentOf t)#0
    and leftOf (contentOf t)#0 === altCallName
    then (inner := contentOf (contentOf t)#1;
        if #inner == 0 then error "empty '{ | } alternation";
        inner#0))

altBranchesOf = t -> (
    if delimiterOf t === "|" and #contentOf t == 2
    then join(altBranchesOf (contentOf t)#0, altBranchesOf (contentOf t)#1)
    else {t})

typedKindOf = t -> (
    if delimiterOf t === spaceOperator and #contentOf t == 2 and isLeaf (contentOf t)#0
    and match("^" | metavarKindPrefix, leftOf (contentOf t)#0)
    then substring(#metavarKindPrefix, leftOf (contentOf t)#0))

markNodes = t -> (
    if isLeaf t then (
        if leftOf t =!= null and match("^" | metavarPlaceholderPrefix, leftOf t)
        then metavarNode substring(#metavarPlaceholderPrefix, leftOf t) else t)
    else if typedKindOf t =!= null then (
        kind := typedKindOf t;
        if not nodeKindNames#?kind then error("unknown node kind in pattern: '" | kind);
        hole := leftOf (contentOf (contentOf t)#1)#0;
        metavarNode(substring(#metavarPlaceholderPrefix, hole), kind))
    else if quantifierOf t =!= null then (
        (sep, elems) := unitOf t;
        repetitionNode(quantifierOf t, sep, apply(elems, markNodes)))
    else if altInnerOf t =!= null then
        alternationNode apply(altBranchesOf altInnerOf t, markNodes)
    else (setContent(t, apply(contentOf t, markNodes));
        t))

templateCache = new CacheTable
parseTemplate = src ->
templateCache#src ??= markNodes parseMacroTree toPlaceholders scanReps src

metavarNamesIn = t -> (
    if isMetavar t then {metavarName t}
    else flatten apply(contentOf t, metavarNamesIn))

treeEquals = (a, b) -> (
    leftOf a === leftOf b and rightOf a === rightOf b and delimiterOf a === delimiterOf b
    and #contentOf a == #contentOf b
    and all(#contentOf a, i -> treeEquals((contentOf a)#i, (contentOf b)#i)))

matchRepetition = (rep, ielems, b) -> (
    unit := repUnit rep;
    u := #unit;
    if u == 0 then error "empty '{ } repetition unit";
    scan(metavarNamesIn rep, nm -> if b#?nm and not instance(b#nm, List) then
            error("metavariable '" | nm | " is bound both outside and inside a repetition"));
    if #ielems % u != 0 then return false;
    nChunks := #ielems // u;
    if repQuantifier rep === "+" and nChunks == 0 then return false;
    ok := all(nChunks, ci -> (
        tb := new MutableHashTable;
        chunkOK := all(u, j -> matchInto(unit#j, ielems#(ci * u + j), tb));
        if chunkOK then scan(keys tb, nm -> b#nm = append(b#nm ?? {}, tb#nm));
        chunkOK));
    if ok and nChunks == 0 then scan(metavarNamesIn rep, nm -> b#nm ??= {});
    ok)

matchElems = (pelems, ielems, b) -> (
    reps := positions(pelems, isRepetition);
    if #reps == 0 then #pelems == #ielems and all(#pelems, i -> matchInto(pelems#i, ielems#i, b))
    else if #reps > 1 then error "a pattern sequence may hold at most one '{ } repetition"
    else (
        r := first reps;
        before := take(pelems, r);
        after := drop(pelems, r + 1);
        if #ielems < #before + #after then return false;
        nRep := #ielems - #before - #after;
        all(#before, i -> matchInto(before#i, ielems#i, b))
        and all(#after, i -> matchInto(after#i, ielems#(#before + nRep + i), b))
        and matchRepetition(pelems#r, take(drop(ielems, #before), nRep), b)))

matchInto = (pat, inp, b) -> (
    if isMetavar pat then (
        if metavarKind pat =!= null and nodeKind inp =!= metavarKind pat then false
        else (
            name := metavarName pat;
            if b#?name then treeEquals(b#name, inp)
            else (b#name = inp;
                true))
    )
    else if isAlternation pat then (
        matched := false;
        for branch in altBranches pat when not matched do (
            tb := new MutableHashTable;
            if matchInto(branch, inp, tb) and all(keys tb, k -> not b#?k or treeEquals(b#k, tb#k))
            then (scan(keys tb, k -> b#k = tb#k);
                matched = true));
        matched
    )
    else if isRepetition pat then
        matchRepetition(pat, if isSeqNode inp then contentOf inp else {inp}, b)
    else if isSeqNode pat and any(contentOf pat, isRepetition) then (
        if isSeqNode inp and delimiterOf pat === delimiterOf inp then matchElems(contentOf pat,
            contentOf inp, b)
        else if isSeqNode inp then false
        else matchElems(contentOf pat, {inp}, b)
    )
    else if #contentOf pat == 1 and isRepetition first contentOf pat
    and leftOf pat === leftOf inp and rightOf pat === rightOf inp
    and delimiterOf pat === delimiterOf inp then
        matchRepetition(first contentOf pat,
        flatten apply(contentOf inp, ic -> if isSeqNode ic then contentOf ic else {ic}), b)
    else if leftOf pat =!= leftOf inp or rightOf pat =!= rightOf inp
    or delimiterOf pat =!= delimiterOf inp
    or #contentOf pat =!= #contentOf inp then false
    else (
        cs := contentOf pat;
        ds := contentOf inp;
        all(#cs, i -> matchInto(cs#i, ds#i, b))
    ))

matchPattern = (pat, inp) -> (
    b := new MutableHashTable;
    if matchInto(pat, inp, b) then new HashTable from b)

cloneTree = t -> (class t)(leftOf t, apply(contentOf t, cloneTree), rightOf t, delimiterOf t)

expandRepetition = (rep, b) -> (
    unit := repUnit rep;
    names := select(metavarNamesIn rep, nm -> b#?nm);
    lengths := unique apply(names, nm -> #(b#nm));
    if #lengths > 1 then error "template repetition metavariables have differing lengths";
    nReps := if #names == 0 then 0 else first lengths;
    flatten apply(nReps, i -> (
        perRep := hashTable apply(names, nm -> (nm, (b#nm)#i));
        apply(unit, u -> instantiate(u, perRep)))))

instantiate = (tmpl, b) -> (
    if isAlternation tmpl then
        error "alternation '{ a | b } is a pattern-only construct, not valid in a template";
    if isMetavar tmpl then (
        name := metavarName tmpl;
        if not b#?name
        then error("template metavariable '" | name | " is unbound");
        cloneTree b#name
    )
    else if any(contentOf tmpl, isRepetition) then (
        if isSeqNode tmpl then
            TokenTree(leftOf tmpl,
            flatten apply(contentOf tmpl, c -> if isRepetition c then expandRepetition(c, b) else {
                instantiate(c, b)}),
            rightOf tmpl, delimiterOf tmpl)
        else (
            if #contentOf tmpl != 1 then
                error "a repetition '{ }+ in a template must be the only content of a sequence or bracket";
            rep := first contentOf tmpl;
            inner := delimited(repSeparator rep, expandRepetition(rep, b));
            TokenTree(leftOf tmpl, {inner}, rightOf tmpl, delimiterOf tmpl))
    )
    else (class tmpl)(leftOf tmpl, apply(contentOf tmpl, c -> instantiate(c, b)), rightOf tmpl,
        delimiterOf tmpl))

quote = method(Dispatch => Thing)
quote String := TokenTree => src -> instantiate(parseTemplate src, new HashTable)
quote Sequence := TokenTree => s -> (
    rest := drop(s, 1);
    binding := if #rest == 1 and instance(first rest, HashTable) then first rest
        else hashTable apply(rest, o -> (toString o#0, o#1));
    instantiate(parseTemplate first s, binding))

matchesIn = method()
matchesIn (TokenTree, TokenTree) := List => (pat, tree) -> (
    below := flatten apply(contentOf tree, c -> matchesIn(pat, c));
    here := matchPattern(pat, tree);
    if here =!= null then prepend((tree, here), below) else below)
patternCell = src -> (
    p := parseTemplate src;
    if delimiterOf p === statementSeparator and #contentOf p == 1 then (contentOf p)#0 else p)
matchesIn (String, TokenTree) := List => (patSrc, tree) -> matchesIn(patternCell patSrc, tree)

expandRules = (name, rules, inp) -> (
    for r in rules do (
        (pat, tmpl) := r;
        b := matchPattern(pat, inp);
        if b =!= null then
            return instantiate(tmpl, b)
    );
    error(name | ": no rule matched the input"))

declMacro = method()

declMacro (String, List) := Macro => (name, rules) -> (
    scan(rules, r -> if not ((instance(r, Sequence) or instance(r, List)) and #r == 2) then
            error(name | ": each rule must be a (pattern, template) pair, got " | toString r));
    parsed := apply(rules, r -> (parseTemplate r#0, parseTemplate r#1));
    installMacro(name, ts -> expandRules(name, parsed, focus ts)))

declMacro (String, String, String) := Macro => (name, p, t) -> declMacro(name, {(p, t)})

Haskell

Algebraic data types, type classes, constrained polymorphism, higher-order folds, pattern matching, comprehensions, and concurrent traversal.

  mdBook
  tree-sitter

{-# LANGUAGE DeriveFunctor #-}
{-# LANGUAGE LambdaCase #-}

module Pipeline where

import Control.Concurrent.Async (mapConcurrently)
import Data.Map.Strict (Map)
import qualified Data.Map.Strict as Map

data Result a
  = Success a
  | Failure String
  deriving (Eq, Show, Functor)

data Job a = Job
  { jobName :: String
  , jobInput :: a
  } deriving (Eq, Show, Functor)

class Runnable task where
  run :: task a -> IO (Result a)

instance Runnable Job where
  run job
    | null (jobName job) =
        pure (Failure "missing name")
    | otherwise =
        pure (Success (jobInput job))

partitionResults
  :: [Result a]
  -> ([String], [a])
partitionResults = foldr step ([], [])
  where
    step result (errors, values) =
      case result of
        Failure message ->
          (message : errors, values)
        Success value ->
          (errors, value : values)

runAll
  :: Runnable task
  => [task a]
  -> IO (Map String a)
runAll jobs = do
  results <- mapConcurrently run jobs
  let (_, values) = partitionResults results
      names = ["job-" <> show n | n <- [1 :: Int ..]]
  pure (Map.fromList (zip names values))

describe :: Result a -> String
describe = \case
  Success _ -> "completed"
  Failure message -> "failed: " <> message


{-# LANGUAGE DeriveFunctor #-}
{-# LANGUAGE LambdaCase #-}

module Pipeline where

import Control.Concurrent.Async (mapConcurrently)
import Data.Map.Strict (Map)
import qualified Data.Map.Strict as Map

data Result a
  = Success a
  | Failure String
  deriving (Eq, Show, Functor)

data Job a = Job
  { jobName :: String
  , jobInput :: a
  } deriving (Eq, Show, Functor)

class Runnable task where
  run :: task a -> IO (Result a)

instance Runnable Job where
  run job
    | null (jobName job) =
        pure (Failure "missing name")
    | otherwise =
        pure (Success (jobInput job))

partitionResults
  :: [Result a]
  -> ([String], [a])
partitionResults = foldr step ([], [])
  where
    step result (errors, values) =
      case result of
        Failure message ->
          (message : errors, values)
        Success value ->
          (errors, value : values)

runAll
  :: Runnable task
  => [task a]
  -> IO (Map String a)
runAll jobs = do
  results <- mapConcurrently run jobs
  let (_, values) = partitionResults results
      names = ["job-" <> show n | n <- [1 :: Int ..]]
  pure (Map.fromList (zip names values))

describe :: Result a -> String
describe = \case
  Success _ -> "completed"
  Failure message -> "failed: " <> message

Lua

Metatables, method syntax, closures, coroutine-based iteration, higher-order functions, table operations, and protected calls.

  mdBook
  tree-sitter

local Stream = {}
Stream.__index = Stream

function Stream.new(source)
  return setmetatable({
    source = source,
    steps = {},
  }, Stream)
end

function Stream:map(fn)
  self.steps[#self.steps + 1] = function(value)
    return true, fn(value)
  end
  return self
end

function Stream:filter(predicate)
  self.steps[#self.steps + 1] = function(value)
    return predicate(value), value
  end
  return self
end

function Stream:iter()
  return coroutine.wrap(function()
    for value in self.source do
      local keep = true
      local current = value
      for _, step in ipairs(self.steps) do
        keep, current = step(current)
        if not keep then break end
      end
      if keep then coroutine.yield(current) end
    end
  end)
end

local function values(items)
  local index = 0
  return function()
    index = index + 1
    return items[index]
  end
end

local stream = Stream.new(values({ 1, 2, 3, 4, 5 }))
  :filter(function(n) return n % 2 == 1 end)
  :map(function(n) return n * n end)

local ok, err = pcall(function()
  for value in stream:iter() do
    print(("square: %d"):format(value))
  end
end)

if not ok then
  io.stderr:write(err, "\n")
end


local Stream = {}
Stream.__index = Stream

function Stream.new(source)
  return setmetatable({
    source = source,
    steps = {},
  }, Stream)
end

function Stream:map(fn)
  self.steps[#self.steps + 1] = function(value)
    return true, fn(value)
  end
  return self
end

function Stream:filter(predicate)
  self.steps[#self.steps + 1] = function(value)
    return predicate(value), value
  end
  return self
end

function Stream:iter()
  return coroutine.wrap(function()
    for value in self.source do
      local keep = true
      local current = value
      for _, step in ipairs(self.steps) do
        keep, current = step(current)
        if not keep then break end
      end
      if keep then coroutine.yield(current) end
    end
  end)
end

local function values(items)
  local index = 0
  return function()
    index = index + 1
    return items[index]
  end
end

local stream = Stream.new(values({ 1, 2, 3, 4, 5 }))
  :filter(function(n) return n % 2 == 1 end)
  :map(function(n) return n * n end)

local ok, err = pcall(function()
  for value in stream:iter() do
    print(("square: %d"):format(value))
  end
end)

if not ok then
  io.stderr:write(err, "\n")
end

Python

Data classes, protocols, generics, async iteration, task groups, comprehensions, enums, and structural pattern matching.

  mdBook
  tree-sitter

from collections.abc import AsyncIterator
from dataclasses import dataclass, field
from enum import StrEnum, auto
from typing import Protocol, TypeVar
import asyncio

T = TypeVar("T")


class State(StrEnum):
    QUEUED = auto()
    RUNNING = auto()
    DONE = auto()
    FAILED = auto()


@dataclass(slots=True)
class Job:
    name: str
    command: tuple[str, ...]
    state: State = State.QUEUED
    output: list[str] = field(default_factory=list)


class Store(Protocol[T]):
    async def save(self, key: str, value: T) -> None: ...
    async def load(self, key: str) -> T | None: ...


async def run(job: Job) -> AsyncIterator[str]:
    job.state = State.RUNNING
    proc = await asyncio.create_subprocess_exec(
        *job.command,
        stdout=asyncio.subprocess.PIPE,
    )
    assert proc.stdout is not None

    async for raw in proc.stdout:
        line = raw.decode().rstrip()
        job.output.append(line)
        yield line

    code = await proc.wait()
    job.state = State.DONE if code == 0 else State.FAILED


async def supervise(
    jobs: list[Job],
    store: Store[Job],
) -> dict[State, int]:
    async with asyncio.TaskGroup() as group:
        for job in jobs:
            group.create_task(store.save(job.name, job))

    counts = {state: 0 for state in State}
    for job in jobs:
        match job.state:
            case State.FAILED if job.output:
                print(f"{job.name}: {job.output[-1]}")
            case State.DONE:
                print(f"{job.name}: complete")
            case _:
                pass
        counts[job.state] += 1
    return counts


from collections.abc import AsyncIterator
from dataclasses import dataclass, field
from enum import StrEnum, auto
from typing import Protocol, TypeVar
import asyncio

T = TypeVar("T")


class State(StrEnum):
    QUEUED = auto()
    RUNNING = auto()
    DONE = auto()
    FAILED = auto()


@dataclass(slots=True)
class Job:
    name: str
    command: tuple[str, ...]
    state: State = State.QUEUED
    output: list[str] = field(default_factory=list)


class Store(Protocol[T]):
    async def save(self, key: str, value: T) -> None: ...
    async def load(self, key: str) -> T | None: ...


async def run(job: Job) -> AsyncIterator[str]:
    job.state = State.RUNNING
    proc = await asyncio.create_subprocess_exec(
        *job.command,
        stdout=asyncio.subprocess.PIPE,
    )
    assert proc.stdout is not None

    async for raw in proc.stdout:
        line = raw.decode().rstrip()
        job.output.append(line)
        yield line

    code = await proc.wait()
    job.state = State.DONE if code == 0 else State.FAILED


async def supervise(
    jobs: list[Job],
    store: Store[Job],
) -> dict[State, int]:
    async with asyncio.TaskGroup() as group:
        for job in jobs:
            group.create_task(store.save(job.name, job))

    counts = {state: 0 for state in State}
    for job in jobs:
        match job.state:
            case State.FAILED if job.output:
                print(f"{job.name}: {job.output[-1]}")
            case State.DONE:
                print(f"{job.name}: complete")
            case _:
                pass
        counts[job.state] += 1
    return counts

C

Enums, tagged unions, slices, function pointers, dynamic storage, designated initializers, and explicit cleanup.

  mdBook
  tree-sitter

#include <stdbool.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>

typedef enum {
    EVENT_DATA,
    EVENT_ERROR,
    EVENT_CLOSED,
} EventKind;

typedef struct {
    const char *data;
    size_t length;
} Slice;

typedef struct {
    EventKind kind;
    union {
        Slice data;
        int error_code;
    };
} Event;

typedef bool (*EventHandler)(
    const Event *event,
    void *context
);

typedef struct {
    Event *items;
    size_t length;
    size_t capacity;
} EventQueue;

static bool queue_push(
    EventQueue *queue,
    Event event
) {
    if (queue->length == queue->capacity) {
        size_t capacity =
            queue->capacity ? queue->capacity * 2 : 8;
        Event *items = realloc(
            queue->items,
            capacity * sizeof(*items)
        );
        if (items == NULL) {
            return false;
        }
        queue->items = items;
        queue->capacity = capacity;
    }

    queue->items[queue->length++] = event;
    return true;
}

static void queue_drain(
    EventQueue *queue,
    EventHandler handle,
    void *context
) {
    for (size_t i = 0; i < queue->length; ++i) {
        if (!handle(&queue->items[i], context)) {
            break;
        }
    }
    queue->length = 0;
}

static bool print_event(
    const Event *event,
    void *context
) {
    FILE *output = context;
    switch (event->kind) {
    case EVENT_DATA:
        fprintf(
            output,
            "%.*s\n",
            (int)event->data.length,
            event->data.data
        );
        return true;
    case EVENT_ERROR:
        fprintf(output, "error %d\n", event->error_code);
        return false;
    case EVENT_CLOSED:
        return false;
    }
    return false;
}

int main(void) {
    EventQueue queue = {0};
    queue_push(&queue, (Event){
        .kind = EVENT_DATA,
        .data = {"ready", 5},
    });
    queue_drain(&queue, print_event, stdout);
    free(queue.items);
}


#include <stdbool.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>

typedef enum {
    EVENT_DATA,
    EVENT_ERROR,
    EVENT_CLOSED,
} EventKind;

typedef struct {
    const char *data;
    size_t length;
} Slice;

typedef struct {
    EventKind kind;
    union {
        Slice data;
        int error_code;
    };
} Event;

typedef bool (*EventHandler)(
    const Event *event,
    void *context
);

typedef struct {
    Event *items;
    size_t length;
    size_t capacity;
} EventQueue;

static bool queue_push(
    EventQueue *queue,
    Event event
) {
    if (queue->length == queue->capacity) {
        size_t capacity =
            queue->capacity ? queue->capacity * 2 : 8;
        Event *items = realloc(
            queue->items,
            capacity * sizeof(*items)
        );
        if (items == NULL) {
            return false;
        }
        queue->items = items;
        queue->capacity = capacity;
    }

    queue->items[queue->length++] = event;
    return true;
}

static void queue_drain(
    EventQueue *queue,
    EventHandler handle,
    void *context
) {
    for (size_t i = 0; i < queue->length; ++i) {
        if (!handle(&queue->items[i], context)) {
            break;
        }
    }
    queue->length = 0;
}

static bool print_event(
    const Event *event,
    void *context
) {
    FILE *output = context;
    switch (event->kind) {
    case EVENT_DATA:
        fprintf(
            output,
            "%.*s\n",
            (int)event->data.length,
            event->data.data
        );
        return true;
    case EVENT_ERROR:
        fprintf(output, "error %d\n", event->error_code);
        return false;
    case EVENT_CLOSED:
        return false;
    }
    return false;
}

int main(void) {
    EventQueue queue = {0};
    queue_push(&queue, (Event){
        .kind = EVENT_DATA,
        .data = {"ready", 5},
    });
    queue_drain(&queue, print_event, stdout);
    free(queue.items);
}

TypeScript

Discriminated unions, generic interfaces, private fields, async generators, type narrowing, and satisfies.

  mdBook
  tree-sitter

type Loading = { state: "loading"; id: string };
type Ready<T> = { state: "ready"; value: T };
type Failed = { state: "failed"; error: Error };
type Result<T> = Loading | Ready<T> | Failed;

interface Cache<T extends { id: string }> {
  get(id: string): Promise<T | undefined>;
  put(value: T): Promise<void>;
}

class MemoryCache<T extends { id: string }>
  implements Cache<T> {
  readonly #items = new Map<string, T>();

  async get(id: string): Promise<T | undefined> {
    return this.#items.get(id);
  }

  async put(value: T): Promise<void> {
    this.#items.set(value.id, value);
  }
}

async function* pages<T>(
  url: URL,
  decode: (raw: unknown) => T[],
): AsyncGenerator<T> {
  let next: URL | undefined = url;

  while (next) {
    const response = await fetch(next);
    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`);
    }

    const body: unknown = await response.json();
    for (const item of decode(body)) {
      yield item;
    }

    const link = response.headers.get("x-next");
    next = link ? new URL(link, next) : undefined;
  }
}

function describe<T>(result: Result<T>): string {
  switch (result.state) {
    case "loading":
      return `loading ${result.id}`;
    case "ready":
      return JSON.stringify(result.value);
    case "failed":
      return result.error.message;
  }
}

const config = {
  retries: 3,
  mode: "eager",
} as const satisfies Record<string, unknown>;


type Loading = { state: "loading"; id: string };
type Ready<T> = { state: "ready"; value: T };
type Failed = { state: "failed"; error: Error };
type Result<T> = Loading | Ready<T> | Failed;

interface Cache<T extends { id: string }> {
  get(id: string): Promise<T | undefined>;
  put(value: T): Promise<void>;
}

class MemoryCache<T extends { id: string }>
  implements Cache<T> {
  readonly #items = new Map<string, T>();

  async get(id: string): Promise<T | undefined> {
    return this.#items.get(id);
  }

  async put(value: T): Promise<void> {
    this.#items.set(value.id, value);
  }
}

async function* pages<T>(
  url: URL,
  decode: (raw: unknown) => T[],
): AsyncGenerator<T> {
  let next: URL | undefined = url;

  while (next) {
    const response = await fetch(next);
    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`);
    }

    const body: unknown = await response.json();
    for (const item of decode(body)) {
      yield item;
    }

    const link = response.headers.get("x-next");
    next = link ? new URL(link, next) : undefined;
  }
}

function describe<T>(result: Result<T>): string {
  switch (result.state) {
    case "loading":
      return `loading ${result.id}`;
    case "ready":
      return JSON.stringify(result.value);
    case "failed":
      return result.error.message;
  }
}

const config = {
  retries: 3,
  mode: "eager",
} as const satisfies Record<string, unknown>;

Bash

Strict mode, arrays, associative maps, traps, process substitution, pattern matching, parameter expansion, and quoting.

  mdBook
  tree-sitter

#!/usr/bin/env bash
set -euo pipefail

declare -A checksums=()
declare -a artifacts=()
workdir="$(mktemp -d)"

cleanup() {
  local status=$?
  rm -rf -- "$workdir"
  exit "$status"
}
trap cleanup EXIT INT TERM

log() {
  printf '[%(%H:%M:%S)T] %s\n' -1 "$*"
}

discover() {
  local root=$1
  while IFS= read -r -d '' path; do
    artifacts+=("$path")
  done < <(
    find "$root" -type f \
      \( -name '*.tar.gz' -o -name '*.zip' \) \
      -print0
  )
}

verify() {
  local path=$1
  local name=${path##*/}
  local expected=${checksums[$name]:-}

  case "$name" in
    *.tar.gz) tar -tzf "$path" >/dev/null ;;
    *.zip) unzip -tq "$path" >/dev/null ;;
    *) return 2 ;;
  esac

  if [[ -n $expected ]]; then
    local actual
    actual=$(sha256sum "$path")
    [[ ${actual%% *} == "$expected" ]]
  fi
}

main() {
  checksums[release.tar.gz]=$(
    cut -d' ' -f1 checksums.txt
  )
  discover "${1:-dist}"

  local failed=0
  for artifact in "${artifacts[@]}"; do
    if verify "$artifact"; then
      log "ok: ${artifact##*/}"
    else
      log "failed: ${artifact##*/}"
      ((failed += 1))
    fi
  done
  return "$failed"
}

main "$@"


#!/usr/bin/env bash
set -euo pipefail

declare -A checksums=()
declare -a artifacts=()
workdir="$(mktemp -d)"

cleanup() {
  local status=$?
  rm -rf -- "$workdir"
  exit "$status"
}
trap cleanup EXIT INT TERM

log() {
  printf '[%(%H:%M:%S)T] %s\n' -1 "$*"
}

discover() {
  local root=$1
  while IFS= read -r -d '' path; do
    artifacts+=("$path")
  done < <(
    find "$root" -type f \
      \( -name '*.tar.gz' -o -name '*.zip' \) \
      -print0
  )
}

verify() {
  local path=$1
  local name=${path##*/}
  local expected=${checksums[$name]:-}

  case "$name" in
    *.tar.gz) tar -tzf "$path" >/dev/null ;;
    *.zip) unzip -tq "$path" >/dev/null ;;
    *) return 2 ;;
  esac

  if [[ -n $expected ]]; then
    local actual
    actual=$(sha256sum "$path")
    [[ ${actual%% *} == "$expected" ]]
  fi
}

main() {
  checksums[release.tar.gz]=$(
    cut -d' ' -f1 checksums.txt
  )
  discover "${1:-dist}"

  local failed=0
  for artifact in "${artifacts[@]}"; do
    if verify "$artifact"; then
      log "ok: ${artifact##*/}"
    else
      log "failed: ${artifact##*/}"
      ((failed += 1))
    fi
  done
  return "$failed"
}

main "$@"

Java

Sealed interfaces, records, validation, streams, guarded pattern matching, switch expressions, and virtual threads.

  mdBook
  tree-sitter

import java.time.Instant;
import java.util.List;
import java.util.concurrent.Executors;

sealed interface Event
    permits Started, Progress, Finished {}

record Started(String task, Instant at)
    implements Event {}

record Progress(String task, int percent)
    implements Event {
    Progress {
        if (percent < 0 || percent > 100) {
            throw new IllegalArgumentException("percent");
        }
    }
}

record Finished(String task, boolean ok)
    implements Event {}

final class EventLog<T extends Event> {
    private final List<T> events;

    EventLog(List<T> events) {
        this.events = List.copyOf(events);
    }

    List<String> messages() {
        return events.stream()
            .map(EventLog::describe)
            .toList();
    }

    private static String describe(Event event) {
        return switch (event) {
            case Started(var task, var at) ->
                "%s started at %s".formatted(task, at);
            case Progress(var task, var percent)
                when percent == 100 ->
                task + " is ready";
            case Progress(var task, var percent) ->
                "%s: %d%%".formatted(task, percent);
            case Finished(var task, var ok) when ok ->
                task + " finished";
            case Finished(var task, var ok) ->
                task + " failed";
        };
    }
}

class Main {
    public static void main(String[] args)
        throws Exception {
        try (var tasks =
                 Executors.newVirtualThreadPerTaskExecutor()) {
            var future = tasks.submit(() ->
                new EventLog<>(List.of(
                    new Started("index", Instant.now()),
                    new Progress("index", 100),
                    new Finished("index", true)
                )).messages()
            );
            future.get().forEach(System.out::println);
        }
    }
}


import java.time.Instant;
import java.util.List;
import java.util.concurrent.Executors;

sealed interface Event
    permits Started, Progress, Finished {}

record Started(String task, Instant at)
    implements Event {}

record Progress(String task, int percent)
    implements Event {
    Progress {
        if (percent < 0 || percent > 100) {
            throw new IllegalArgumentException("percent");
        }
    }
}

record Finished(String task, boolean ok)
    implements Event {}

final class EventLog<T extends Event> {
    private final List<T> events;

    EventLog(List<T> events) {
        this.events = List.copyOf(events);
    }

    List<String> messages() {
        return events.stream()
            .map(EventLog::describe)
            .toList();
    }

    private static String describe(Event event) {
        return switch (event) {
            case Started(var task, var at) ->
                "%s started at %s".formatted(task, at);
            case Progress(var task, var percent)
                when percent == 100 ->
                task + " is ready";
            case Progress(var task, var percent) ->
                "%s: %d%%".formatted(task, percent);
            case Finished(var task, var ok) when ok ->
                task + " finished";
            case Finished(var task, var ok) ->
                task + " failed";
        };
    }
}

class Main {
    public static void main(String[] args)
        throws Exception {
        try (var tasks =
                 Executors.newVirtualThreadPerTaskExecutor()) {
            var future = tasks.submit(() ->
                new EventLog<>(List.of(
                    new Started("index", Instant.now()),
                    new Progress("index", 100),
                    new Finished("index", true)
                )).messages()
            );
            future.get().forEach(System.out::println);
        }
    }
}

PHP

Attributes, enums, readonly classes, interfaces, generators, named arguments, arrow functions, and match.

  mdBook
  tree-sitter

<?php

declare(strict_types=1);

namespace App\Jobs;

use Attribute;
use Generator;

#[Attribute(Attribute::TARGET_CLASS)]
final readonly class Queue
{
    public function __construct(
        public string $name,
        public int $retries = 3,
    ) {}
}

enum State: string
{
    case Queued = 'queued';
    case Running = 'running';
    case Done = 'done';
    case Failed = 'failed';

    public function terminal(): bool
    {
        return match ($this) {
            self::Done, self::Failed => true,
            default => false,
        };
    }
}

interface Job
{
    public function id(): string;
    public function run(Context $ctx): State;
}

#[Queue('reports', retries: 5)]
final readonly class BuildReport implements Job
{
    public function __construct(
        private string $reportId,
        private array $records,
    ) {}

    public function id(): string
    {
        return $this->reportId;
    }

    public function rows(): Generator
    {
        foreach ($this->records as $record) {
            yield [
                'name' => $record['name'],
                'score' => (float) $record['score'],
            ];
        }
    }

    public function run(Context $ctx): State
    {
        $rows = [...$this->rows()];
        usort(
            $rows,
            fn(array $a, array $b): int =>
                $b['score'] <=> $a['score'],
        );

        return $ctx->write($this->id(), $rows)
            ? State::Done
            : State::Failed;
    }
}


<?php

declare(strict_types=1);

namespace App\Jobs;

use Attribute;
use Generator;

#[Attribute(Attribute::TARGET_CLASS)]
final readonly class Queue
{
    public function __construct(
        public string $name,
        public int $retries = 3,
    ) {}
}

enum State: string
{
    case Queued = 'queued';
    case Running = 'running';
    case Done = 'done';
    case Failed = 'failed';

    public function terminal(): bool
    {
        return match ($this) {
            self::Done, self::Failed => true,
            default => false,
        };
    }
}

interface Job
{
    public function id(): string;
    public function run(Context $ctx): State;
}

#[Queue('reports', retries: 5)]
final readonly class BuildReport implements Job
{
    public function __construct(
        private string $reportId,
        private array $records,
    ) {}

    public function id(): string
    {
        return $this->reportId;
    }

    public function rows(): Generator
    {
        foreach ($this->records as $record) {
            yield [
                'name' => $record['name'],
                'score' => (float) $record['score'],
            ];
        }
    }

    public function run(Context $ctx): State
    {
        $rows = [...$this->rows()];
        usort(
            $rows,
            fn(array $a, array $b): int =>
                $b['score'] <=> $a['score'],
        );

        return $ctx->write($this->id(), $rows)
            ? State::Done
            : State::Failed;
    }
}

Go

Type constraints, generic interfaces, goroutines, channels, select, contexts, and error handling.

  mdBook
  tree-sitter

package pipeline

import (
	"context"
	"errors"
	"sync"
)

type Item interface {
	Key() string
}

type Store[T Item] interface {
	Load(context.Context, string) (T, error)
	Save(context.Context, T) error
}

type Result[T any] struct {
	Value T
	Err   error
}

func Map[A, B any](
	ctx context.Context,
	input <-chan A,
	workers int,
	fn func(context.Context, A) (B, error),
) <-chan Result[B] {
	output := make(chan Result[B])
	var group sync.WaitGroup

	group.Add(workers)
	for range workers {
		go func() {
			defer group.Done()
			for value := range input {
				item, err := fn(ctx, value)
				select {
				case output <- Result[B]{
					Value: item,
					Err:   err,
				}:
				case <-ctx.Done():
					return
				}
			}
		}()
	}

	go func() {
		group.Wait()
		close(output)
	}()
	return output
}

func Collect[T any](
	ctx context.Context,
	input <-chan Result[T],
) ([]T, error) {
	var values []T
	for {
		select {
		case <-ctx.Done():
			return nil, ctx.Err()
		case result, ok := <-input:
			if !ok {
				return values, nil
			}
			if result.Err != nil {
				return nil, errors.Join(
					result.Err,
					ctx.Err(),
				)
			}
			values = append(values, result.Value)
		}
	}
}


package pipeline

import (
	"context"
	"errors"
	"sync"
)

type Item interface {
	Key() string
}

type Store[T Item] interface {
	Load(context.Context, string) (T, error)
	Save(context.Context, T) error
}

type Result[T any] struct {
	Value T
	Err   error
}

func Map[A, B any](
	ctx context.Context,
	input <-chan A,
	workers int,
	fn func(context.Context, A) (B, error),
) <-chan Result[B] {
	output := make(chan Result[B])
	var group sync.WaitGroup

	group.Add(workers)
	for range workers {
		go func() {
			defer group.Done()
			for value := range input {
				item, err := fn(ctx, value)
				select {
				case output <- Result[B]{
					Value: item,
					Err:   err,
				}:
				case <-ctx.Done():
					return
				}
			}
		}()
	}

	go func() {
		group.Wait()
		close(output)
	}()
	return output
}

func Collect[T any](
	ctx context.Context,
	input <-chan Result[T],
) ([]T, error) {
	var values []T
	for {
		select {
		case <-ctx.Done():
			return nil, ctx.Err()
		case result, ok := <-input:
			if !ok {
				return values, nil
			}
			if result.Err != nil {
				return nil, errors.Join(
					result.Err,
					ctx.Err(),
				)
			}
			values = append(values, result.Value)
		}
	}
}

JavaScript

Private fields, closures, destructuring, optional chaining, promises, and async generators.

  mdBook
  tree-sitter

class EventBus {
  #listeners = new Map();

  on(type, listener) {
    const listeners = this.#listeners.get(type) ?? [];
    listeners.push(listener);
    this.#listeners.set(type, listeners);
    return () => this.off(type, listener);
  }

  off(type, listener) {
    const listeners = this.#listeners.get(type);
    this.#listeners.set(
      type,
      listeners?.filter(item => item !== listener) ?? [],
    );
  }

  async emit(type, detail) {
    const listeners = this.#listeners.get(type) ?? [];
    const event = { type, detail, time: Date.now() };
    return Promise.all(
      listeners.map(listener => listener(event)),
    );
  }
}

async function* readPages(start, { signal } = {}) {
  let url = new URL(start);

  while (url) {
    const response = await fetch(url, { signal });
    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`);
    }

    const { items, next } = await response.json();
    yield* items;
    url = next ? new URL(next, url) : null;
  }
}

const bus = new EventBus();
const stop = bus.on("item", ({ detail }) => {
  console.log(detail?.name ?? "unnamed");
});

for await (const item of readPages("/api/items")) {
  await bus.emit("item", item);
}
stop();


class EventBus {
  #listeners = new Map();

  on(type, listener) {
    const listeners = this.#listeners.get(type) ?? [];
    listeners.push(listener);
    this.#listeners.set(type, listeners);
    return () => this.off(type, listener);
  }

  off(type, listener) {
    const listeners = this.#listeners.get(type);
    this.#listeners.set(
      type,
      listeners?.filter(item => item !== listener) ?? [],
    );
  }

  async emit(type, detail) {
    const listeners = this.#listeners.get(type) ?? [];
    const event = { type, detail, time: Date.now() };
    return Promise.all(
      listeners.map(listener => listener(event)),
    );
  }
}

async function* readPages(start, { signal } = {}) {
  let url = new URL(start);

  while (url) {
    const response = await fetch(url, { signal });
    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`);
    }

    const { items, next } = await response.json();
    yield* items;
    url = next ? new URL(next, url) : null;
  }
}

const bus = new EventBus();
const stop = bus.on("item", ({ detail }) => {
  console.log(detail?.name ?? "unnamed");
});

for await (const item of readPages("/api/items")) {
  await bus.emit("item", item);
}
stop();

Contributing

Thanks for your interest in mdbook-tsitter. The crate is deliberately small and grammar-agnostic — it ships no grammar of its own and highlights whatever you configure in book.toml.

Building

cargo build
cargo clippy --all
cargo fmt

Clippy should be clean and cargo fmt applied before a change is considered done. Public items carry doc comments.

Running the example

The book under examples/languages doubles as the documentation site and as an integration test across several grammars. Grammars are external (compiled parsers + third-party queries), so they are staged locally rather than committed:

cd examples/languages
./setup.sh            # stage parsers/ and queries/ (gitignored)
mdbook build          # needs the mdbook-tsitter binary on PATH

setup.sh copies parsers and queries from a local nvim-treesitter install by default; override the source paths with the environment variables documented at the top of the script.

Scope

The package stays language-agnostic: please do not add grammars or language-specific behaviour to the crate itself. New languages belong in a book’s configuration (and, for the example, in setup.sh).

Pull requests

Keep changes focused and the commit history readable.
Match the surrounding code’s style and comment density.
Update the README and the example when behaviour or configuration changes.

Keyboard shortcuts

mdbook-tsitter