mdbook-tsitter

Tree-sitter syntax highlighting for mdBook — for any language you can point a grammar at.

View the project on GitHub.

mdbook-tsitter is an mdBook preprocessor that parses fenced code blocks with tree-sitter and renders the resulting captures as themeable HTML. It is grammar-agnostic: list a language it knows, name any grammar’s git repository, or supply a compiled parser and its highlight queries, and the same preprocessor can handle anything from Rust and Lua to a language of your own.

Features

• Grammar-agnostic. Over 300 common languages work by name; any other tree-sitter grammar works from its repository or a prebuilt parser. Grammars are fetched and compiled on the first build.
• Structural highlighting. Definitions, calls, types, parameters, properties, macros, and other syntax can receive distinct styles whenever the grammar’s queries distinguish them.
• Embedded languages. Injection queries can highlight fenced code inside Markdown, languages embedded in strings, and other nested syntax.
• Themeable capture classes. Captures become predictable ts-… CSS classes, with a bundled stylesheet that follows mdBook’s light, rust, coal, navy, and ayu themes. A highlight_ts.css in the theme directory replaces it, and a theme that only restyles highlight.js colours tree-sitter blocks to match.
• Side-by-side blocks. Label consecutive blocks with pane="…" to show them side by side under their labels, such as before and after a change, or the same idea in several languages.
• Language aliases and locals queries. Match several fence names to one grammar and opt into scope-aware query behavior where the grammar supports it.
• Selective processing. Leave individual blocks to mdBook when you need its built-in Rust Playground behavior or other native code-block features.
• Build-time rendering. Highlighted HTML is generated with the book; the reader does not need a tree-sitter runtime.
• Plays well with other preprocessors. Blocks are found exactly as mdBook finds them, and highlighted code stays out of reach of math preprocessors.

The documentation book is built with the preprocessor itself. Its examples include Macaulay2, a grammar outside nvim-treesitter’s list, and substantial side-by-side comparisons in Rust, Python, TypeScript, JavaScript, Go, Java, C, C++, Bash, PHP, Lua, and Haskell, and its book.toml is a complete configuration example.

Installation

Install the preprocessor from crates.io:

cargo install mdbook-tsitter

The mdbook-tsitter binary must be available on PATH when mdbook build runs. Grammars that are fetched rather than prebuilt also need git and a C compiler (cc, gcc, clang, or MSVC) on the build machine; see Grammar sources.

Quick start

List the languages to highlight in book.toml:

[preprocessor.tsitter]

[preprocessor.tsitter.languages]
rust = {}
python = {}
typescript = { aliases = ["ts"] }

The first build fetches and compiles each grammar; later builds reuse it. After that, ordinary fenced blocks for the configured languages are highlighted automatically:

```rust
fn main() {
    println!("highlighted at build time");
}
```

Build the book as usual:

mdbook build

Languages that are not configured, blocks without a language tag, and blocks that opt out are left untouched for mdBook to handle.

Configuring languages

Everything lives under [preprocessor.tsitter].

Preprocessor options

Language options

Add one entry to [preprocessor.tsitter.languages] per grammar. A language from the built-in list needs no keys at all; any other grammar needs git as its source. Paths are relative to the book root.

A language may need its own table when it carries several keys:

[preprocessor.tsitter.languages.markdown]
aliases = ["md", "markdown"]
injections = "queries/markdown/injections.scm"

Injection queries can select another configured grammar using standard @injection.language and @injection.content captures. All grammars share the same capture-class table during a build, so embedded and host languages use the same theme consistently.

Rust Playground and hidden lines

Highlighted blocks keep mdBook’s code-block features. Rust blocks stay runnable in the Rust Playground, with the book’s [rust] edition, the implicit fn main wrapper, and annotations such as ignore, noplayground, mdbook-runnable, or edition2021 handled as mdBook handles them. Hidden lines (# in Rust, or a prefix from hidelines= or [output.html.code.hidelines]) are collapsed behind mdBook’s toggle:

```rust
# use std::collections::HashMap;
let mut scores = HashMap::new();
scores.insert("tree-sitter", 1);
```

Editable playgrounds (editable with [output.html.playground] editable = true) are replaced by mdBook’s in-page editor, so they are always left to mdBook.

Leaving a block to mdBook

Processing a block replaces its Markdown with highlighted HTML. Add the notreesitter annotation when a block should keep mdBook’s highlight.js highlighting instead:

```rust,notreesitter
let highlighted = "by mdBook";
```

Side-by-side blocks

Blocks labelled with pane="…" are shown side by side, and compare shows one block as tree-sitter and highlight.js highlight it. See Side-by-side blocks.

Alongside other preprocessors

mdBook runs preprocessors in the order set by their before and after keys, and otherwise alphabetically by name. A highlighted block becomes HTML, so a preprocessor that generates code blocks, or that must see a block as a code fence, has to run before this one:

[preprocessor.tsitter]
after = ["links", "my-code-generator"]

Blocks are found with mdBook’s own Markdown settings, so a block is highlighted exactly when mdBook would render it as a code block. In highlighted code, $ and \ are written as HTML entities, so math preprocessors such as mdbook-katex leave the code alone whichever of the two runs first.

Grammar sources

From the built-in list

The preprocessor knows the repositories of the grammars nvim-treesitter supports, each pinned to the revision nvim-treesitter tests, and of a few grammars nvim-treesitter does not list, such as macaulay2, each pinned to a release. Use the language’s name there as its entry; - and _ are interchangeable (c-sharp or c_sharp). A mistyped name fails with a suggestion when a listed name is close.

Their queries are nvim-treesitter’s for the pinned revision, with the queries they inherit from other languages; they are tested against exactly that parser and are usually richer than a repository’s own. Where nvim-treesitter has none, or they do not match the parser, the repository’s own queries are used.

From a git repository

With git, the repository is shallow-fetched at rev and its parser (src/parser.c plus any external scanner) is compiled with the platform’s C compiler. A grammar that commits no generated parser is generated first, which needs the tree-sitter CLI and Node.js.

The repository’s tree-sitter.json locates the grammar and lists its queries, so location is rarely needed. Queries a grammar builds on from another grammar (TypeScript’s extend JavaScript’s) are fetched from that grammar’s repository, at the version the grammar’s package.json pins. Without a tree-sitter.json, queries are read from queries/ beside the grammar or at the repository root.

Checkouts and compiled parsers are cached per user, under mdbook-tsitter/ in the user cache directory ($XDG_CACHE_HOME or ~/.cache on Linux and macOS, %LOCALAPPDATA% on Windows), and shared between books. Network access is only needed the first time a revision is used.

A cached checkout is reused for as long as it exists, so an unpinned grammar stays at the commit first fetched. Pin rev to choose the version explicitly; changing it fetches the new revision. Deleting the cache directory refetches everything.

git may also be a path to a local repository, absolute or relative to the book root, such as a grammar you are developing. Its rev (by default HEAD) is resolved on every build, so new commits are picked up; uncommitted changes are not.

[preprocessor.tsitter.languages.mylang]
git = "../tree-sitter-mylang"

To use different queries, such as your own, set highlights (and optionally injections and locals) to query files in your book.

From a compiled parser (deprecated)

A language may still name a prebuilt parser shared library with library (and its constructor with symbol), with highlights pointing at its query. This is deprecated and prints a warning; see Upgrading from 2.x to replace it.

Theming

A bundled stylesheet is embedded automatically. It covers standard tree-sitter and nvim-treesitter capture names and follows mdBook’s built-in colour schemes, so most books need no CSS of their own.

The embedded stylesheet sits in the mdbook-tsitter cascade layer, so any rule in your own [output.html] additional-css overrides it, whatever its specificity. Add a stylesheet only for what you want to change, such as custom capture names or colours for a custom mdBook theme:

[output.html]
additional-css = ["theme/highlighting.css"]

Each capture gets the ts- prefix, dots become hyphens, and every prefix is emitted so broad rules can cascade into more specific ones:

For example:

.ts-comment {
  font-style: italic;
}

code[data-language="rust"] .ts-function-macro {
  color: var(--ts-purple);
}

Each highlighted block carries its fence tag as data-language, so a code[data-language="…"] selector scopes a rule to one language.

Theme directory

Like mdBook’s own theme files, the highlighting stylesheet can be replaced from the book’s theme directory (theme/, or [output.html] theme). A highlight_ts.css there is embedded instead of the bundled stylesheet, as mdBook uses a highlight.css there instead of its highlight.js theme. Start from the bundled one:

mdbook-tsitter css > theme/highlight_ts.css

Without a highlight_ts.css, a theme that only replaces mdBook’s highlight.js stylesheets still colours tree-sitter blocks: the bundled stylesheet takes each role’s colour from the matching highlight.js class (.hljs-keyword for keywords, .hljs-string for strings, and so on). A replaced highlight.css applies to the light, rust, and custom themes, tomorrow-night.css to coal and navy, and ayu-highlight.css to ayu, as in mdBook.

Set default-css = false under [preprocessor.tsitter] to embed no highlighting stylesheet at all.

There is no fixed capture list in the preprocessor. Capture names come directly from each grammar’s queries; names beginning with _ are treated as internal and left unstyled.

Example: a custom mdBook theme

This book adds Everforest to mdBook’s theme picker; choose it from the brush icon in the menu bar. It is added with mdBook’s theme overrides, and mdBook sets the chosen theme’s name as a class on <html>, so everything the theme defines is scoped to .everforest.

Template

theme/index.hbs is mdBook’s own template, written out with mdbook init --theme, with one entry added to the theme list:

<li role="none"><button role="menuitem" class="theme" id="mdbook-theme-everforest">Everforest</button></li>

mdBook uses any file in theme/ named like one of its own in place of it. Since this one replaces mdBook’s template, refresh it from mdbook init --theme when upgrading mdBook.

Colours

theme/everforest.css sets mdBook’s interface variables, the same ones the built-in themes set in css/variables.css, and the highlighting colours: Everforest’s palette as the hues, and every colour role mapped to the hue Everforest gives that kind of syntax, so none of the bundled stylesheet’s choices remain:

.everforest {
    /* Tree-sitter hues: Everforest's palette. */
    --ts-red:    #e67e80;
    --ts-orange: #e69875;
    --ts-yellow: #dbbc7f;
    --ts-green:  #a7c080;
    --ts-cyan:   #83c092;
    --ts-blue:   #7fbbb3;
    --ts-purple: #d699b6;
    --ts-gray:   #859289;

    /* Every colour role, as Everforest colours that kind of syntax. The
       highlight.js rules below use them; tree-sitter captures are styled
       individually at the end of this file. */
    --ts-comment:     var(--ts-gray);
    --ts-keyword:     var(--ts-red);
    --ts-string:      var(--ts-cyan);
    --ts-escape:      var(--ts-green);
    --ts-number:      var(--ts-purple);
    --ts-function:    var(--ts-green);
    --ts-declaration: var(--ts-green);
    --ts-type:        var(--ts-yellow);
    --ts-special:     var(--ts-orange);
    --ts-namespace:   var(--ts-yellow);
    --ts-attribute:   var(--ts-purple);
    --ts-heading:     var(--ts-orange);
    --ts-link:        var(--ts-blue);
}

Beyond the roles, every tree-sitter capture is styled as Everforest’s Neovim colour scheme highlights it: each @capture group’s colour, italic, bold, and underline, such as italic comments, bold headings, and underlined URLs. The same file styles highlight.js classes in the same roles, so blocks left to mdBook match the tree-sitter ones.

Code editors

mdBook gives editable blocks its light editor theme for any theme it does not know; a short theme/everforest.js switches them to the dark one.

The stylesheet and the script are registered in book.toml:

[output.html]
additional-css = ["theme/comparison.css", "theme/everforest.css"]
additional-js = ["theme/everforest.js"]

Interactive blocks

Highlighted blocks keep mdBook’s own code-block features. Hover a block to see its buttons: copy, run in the Rust Playground, and show hidden lines.

Runnable Rust

A plain rust block runs in the playground with the book’s edition. It has no fn main, so the hidden wrapper mdBook adds is revealed by the eye button.

#![allow(unused)]
fn main() {
let primes: Vec<u32> = (2..50)
    .filter(|n| (2..*n).take_while(|d| d * d <= *n).all(|d| n % d != 0))
    .collect();
println!("{primes:?}");
}

Hidden lines

Lines starting with # are compiled and run but hidden from readers until they are revealed.

#![allow(unused)]
fn main() {
use std::collections::BTreeMap;

fn word_counts(text: &str) -> BTreeMap<&str, usize> {
    let mut counts = BTreeMap::new();
    for word in text.split_whitespace() {
        *counts.entry(word).or_insert(0) += 1;
    }
    counts
}

let counts = word_counts("the quick fox jumps over the lazy dog the end");
for (word, count) in &counts {
    println!("{word:>5}: {count}");
}
}

Annotations

Annotations work as they do in mdBook. should_panic keeps the Run button and the panic shows up in the output:

#![allow(unused)]
fn main() {
let readings = [21.5, 22.0, f64::NAN];
let average = readings.iter().sum::<f64>() / readings.len() as f64;
assert!(!average.is_nan(), "a reading is missing");
}

noplayground keeps the highlighting but removes the Run button:

// Reads from stdin, which the playground cannot provide.
let mut line = String::new();
std::io::stdin().read_line(&mut line)?;

ignore marks code that is not meant to compile:

let config = load_config()?; // defined elsewhere
serve(config.address).await;

Hidden lines in other languages

Other languages hide lines with a prefix of your choice, given per block with hidelines=<prefix> or per language in [output.html.code.hidelines]. This TypeScript block hides its imports and helper behind ~:

import { readFileSync } from "node:fs";

function parse(source: string): Record<string, string> {
  return Object.fromEntries(
    source.split("\n").filter(Boolean).map((line) => line.split("=", 2)),
  );
}

const env = parse(readFileSync(".env", "utf8"));
console.log(`connecting to ${env.HOST}:${env.PORT}`);

This book sets python = "~~" in [output.html.code.hidelines]:

from dataclasses import dataclass

@dataclass
class Point:
    x: float
    y: float

    def norm(self) -> float:
        return (self.x**2 + self.y**2) ** 0.5

print(Point(3, 4).norm())

Editable playgrounds

With editable = true under [output.html.playground], an editable block becomes an in-page editor. mdBook replaces it with its own editor, so it is left to mdBook rather than highlighted by tree-sitter:

fn main() {
    let name = "tree-sitter";
    println!("Hello from {name}! Edit me and press run.");
}

Side-by-side blocks

Code blocks can be shown next to each other, each under a title: a change before and after, the same idea in several languages, or one block as tree-sitter and highlight.js highlight it.

Panes

Give each block a pane annotation holding its title. Blocks with a pane annotation that follow each other, separated only by blank lines, form one group:

```rust,pane="Before"
let total = a + b;
```

```rust,pane="After"
let total = a.saturating_add(b);
```

The group is shown as equal columns, each under its title, centred above the block. On screens narrower than 760 pixels the panes stack instead.

• Titles. Quote a title that contains spaces or commas, as in pane="Before, as written"; a single word needs no quotes, as in pane=Before. A title is plain text, shown exactly as written.
• Any number of panes. A group can hold two blocks or more, each in its own language.
• Each pane renders as its block would on its own. A block in a configured language is highlighted with tree-sitter. A block marked notreesitter, in a language that is not configured, or in an editable playground is left to mdBook. Other annotations keep their meaning, so Rust panes keep their Run button and hidden lines.
• Ending a group. Anything other than a blank line between two blocks, such as a paragraph, a heading, or an HTML comment (<!-- -->), ends the group, so two groups can follow each other.
• Top-level blocks only. Like highlighting, panes apply to fences at the start of a line. A block inside a list item or a blockquote is left to mdBook as written.

Comparing with highlight.js

The compare annotation shows one block twice: on the left highlighted with tree-sitter, on the right as mdBook highlights it with highlight.js.

```rust,compare
fn main() {
    println!("side by side");
}
```

It is a shorthand for two panes from the same code:

```rust,pane="tree-sitter"
fn main() {
    println!("side by side");
}
```

```rust,notreesitter,pane="highlight.js"
fn main() {
    println!("side by side");
}
```

Styling

A group becomes the following HTML, with the layout in the embedded stylesheet:

<div class="treesitter-side-by-side">
  <div class="treesitter-pane">
    <div class="treesitter-pane-label">Before</div>
    <pre>…</pre>
  </div>
  …
</div>

Rules in additional-css override the layout, for example to widen the gap between panes or align titles to the left:

.treesitter-side-by-side { column-gap: 2.5rem; }
.treesitter-pane-label { text-align: left; }

A highlight_ts.css in the theme directory replaces the embedded stylesheet, layout included, so a replacement that is started from mdbook-tsitter css keeps it.

Examples

Before and after

The two panes from Panes, with the change in full:

#![allow(unused)]
fn main() {
fn total_length(words: &[String]) -> usize {
    let mut total = 0;
    for i in 0..words.len() {
        total += words[i].len();
    }
    total
}
}

#![allow(unused)]
fn main() {
fn total_length(words: &[String]) -> usize {
    words.iter().map(String::len).sum()
}
}

One idea, several languages

Three panes, each in its own language:

def sum_squares(xs):
    return sum(x * x for x in xs)

func sumSquares(xs []int) int {
	total := 0
	for _, x := range xs {
		total += x * x
	}
	return total
}

sumSquares :: [Int] -> Int
sumSquares = sum . map (^ 2)

Comparing with highlight.js

A compare block. The language chapters show longer comparisons, written as two panes each:

type Shape =
  | { kind: "circle"; radius: number }
  | { kind: "square"; side: number };

const area = (shape: Shape): number =>
  shape.kind === "circle"
    ? Math.PI * shape.radius ** 2
    : shape.side ** 2;

type Shape =
  | { kind: "circle"; radius: number }
  | { kind: "square"; side: number };

const area = (shape: Shape): number =>
  shape.kind === "circle"
    ? Math.PI * shape.radius ** 2
    : shape.side ** 2;

Upgrading from 2.x

Configurations written for 2.x keep working, and each deprecated setting prints a warning naming its replacement. The deprecated settings will be removed in the next major release.

• library and symbol. For a language from the built-in list, remove them along with the highlights path: the grammar and its queries are fetched instead. Any other grammar is named by git, its repository URL or a path to a local checkout.
• A copy of the bundled stylesheet in additional-css. The stylesheet is now embedded, so remove the copy, or keep only your own changes in it. A copy that replaces the bundled theme moves to theme/highlight_ts.css.

A 2.x configuration such as

[preprocessor.tsitter.languages.rust]
library = "parsers/rust.so"
highlights = "queries/rust/highlights.scm"

[preprocessor.tsitter.languages.mylang]
library = "parsers/mylang.so"
highlights = "queries/mylang/highlights.scm"

[output.html]
additional-css = ["theme/treesitter.css"]

becomes

[preprocessor.tsitter.languages]
rust = {}

[preprocessor.tsitter.languages.mylang]
git = "https://github.com/someone/tree-sitter-mylang"
rev = "v1.0.0"

and the committed parsers, queries, and stylesheet can be deleted.

Highlighting from nvim-treesitter style queries is also more faithful in 3.0: #contains? and Vim’s \c regex flag are evaluated, and spell-checking and helper captures no longer strip the colour from the nodes they mark.

How it works

mdBook passes every chapter to the preprocessor as Markdown. mdbook-tsitter parses it with mdBook’s own Markdown parser and settings (pulldown-cmark with the extensions [output.html] enables), locates top-level fenced code blocks for configured languages, and sends their source through tree-sitter-highlight.

The highlighted events become semantic CSS classes and are spliced back into the chapter as ready-made HTML:

<pre class="treesitter"><code class="no-highlight" data-language="rust">…spans…</code></pre>

The block deliberately has no language-* class: mdBook’s highlight.js would re-highlight any block whose language-* class names a language it knows, even one marked no-highlight, and its colours would override the tree-sitter ones. The fence tag is kept in data-language instead.

mdBook adds its own code-block features (Rust Playground markup, the implicit fn main, hidden lines) only to plain-text blocks, so the preprocessor applies the same rules before highlighting. A runnable Rust block, for example, becomes:

<pre class="treesitter playground"><code class="no-highlight edition2021" data-language="rust">…spans…</code></pre>

Other preprocessors that run afterwards see highlighted blocks as HTML rather than code fences, so $ and \ in highlighted code are written as HTML entities, which math preprocessors do not read as their syntax.

A group of pane="…" blocks becomes a treesitter-side-by-side element with one treesitter-pane per block. A pane holding a block left to mdBook keeps its fence, separated from the surrounding HTML by blank lines so mdBook still renders it as Markdown:

<div class="treesitter-side-by-side">
<div class="treesitter-pane">
<div class="treesitter-pane-label">Before</div>

<pre class="treesitter">…</pre>

</div>
…
</div>

Finally, every chapter with a highlighted block gets the highlighting stylesheet as a <style> element in the mdbook-tsitter cascade layer: the theme directory’s highlight_ts.css, or else the bundled stylesheet, with any colours carried over from highlight.js stylesheets the theme replaces.

Languages

Each chapter in this section renders the same source twice, side by side: on the left as mdbook-tsitter highlights it with tree-sitter, on the right as mdBook highlights it with highlight.js. Macaulay2 has a single tree-sitter example, as highlight.js does not know the language.

Every language here comes from the built-in grammar list and is configured by name alone, Macaulay2 included: its grammar is not in nvim-treesitter’s list, so the built-in list adds it. This is the book’s complete configuration:

[book]
title = "mdbook-tsitter"
src = "src"

# `after = ["links"]` runs this after mdBook's built-in include preprocessor, so
# any configured-language block inside content pulled in via {{#include}} (the
# root README.md / CONTRIBUTING.md) is highlighted too.
[preprocessor.tsitter]
after = ["links"]
# This book documents the preprocessor in this repository, so it runs it from
# source; mdBook starts the command in the book's directory, without a shell.
# A book using the published crate needs no `command`.
command = "cargo run --quiet --manifest-path ../../Cargo.toml --"
# inject = true        # (default) highlight languages embedded via injections
# default-css = true   # (default) embed the bundled stylesheet

# Languages from the built-in list need no source: each grammar is fetched at
# its pinned revision and compiled on the first build, then cached.
[preprocessor.tsitter.languages]
rust = {}
python = {}
typescript = { aliases = ["typescript", "ts"] }
javascript = { aliases = ["javascript", "js"] }
go = {}
java = {}
c = {}
cpp = { aliases = ["cpp", "c++"] }
bash = { aliases = ["bash", "sh", "shell"] }
php = {}
lua = {}
haskell = {}
markdown = { aliases = ["markdown", "md"] }
toml = {}
macaulay2 = { aliases = ["macaulay2", "m2"] }

[rust]
edition = "2021"

[output.html]
site-url = "/mdbook-tsitter/"
# Layout for the side-by-side comparison chapters; highlighting colours come
# from the embedded stylesheet. Everforest is an extra mdBook theme, added to
# the theme picker by `theme/index.hbs` (see the Theming chapter).
additional-css = ["theme/comparison.css", "theme/everforest.css"]
additional-js = ["theme/everforest.js"]

# Playground features kept on highlighted blocks (see the Interactive chapter).
[output.html.playground]
editable = true

[output.html.code.hidelines]
python = "~~"

Rust

Traits, generic bounds, Arc delegation, concurrent maps, locks, iterators, closures, and let-else.

#![allow(unused)]
fn main() {
//! A Rust example of a language that uses the tree-sitter-highlight preprocessor.
use dashmap::DashMap;
use lsp_types::{Location, Range, Url};
use std::{path::PathBuf, sync::{Arc, RwLock}};


#[derive(Debug, Clone)]
struct DefSite {
    uri: Url,
    range: Range,
    kind: TokenKind,
}
/// A symbol name.
/// # Examples
/// ```
/// use treesitter_highlight::{Symbol, Symbol::new};
/// let symbol = new("foo");
/// assert_eq!(symbol.name(), "foo");
/// ```
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct Symbol(String);

impl Symbol {
    pub fn new(name: &str) -> Self {
        Self(name.to_string())
    }

    pub fn name(&self) -> &str {
        &self.0
    }
}
trait DefLookup {
    fn find(&self, name: &str, skip: &Url) -> Vec<Location>;
    fn has(&self, name: &str) -> bool; // Comment
    fn kind(&self, name: &str, skip: &Url) -> Option<TokenKind>;
}

impl<T: DefLookup + ?Sized> DefLookup for Arc<T> {
    fn find(
        &self,
        name: &str,
        skip: &Url,
    ) -> Vec<Location> {
        self.as_ref().find(name, skip)
    }

    fn has(&self, name: &str) -> bool {
        self.as_ref().has(name)
    }

    fn kind(
        &self,
        name: &str,
        skip: &Url,
    ) -> Option<TokenKind> {
        self.as_ref().kind(name, skip)
    }
}

/// A symbol index kept in sync with editor changes.
#[derive(Debug, Default)]
struct SymbolIndex {
    defs: DashMap<Symbol, Vec<DefSite>>,
    files: DashMap<Url, Vec<Symbol>>,
    roots: RwLock<Vec<PathBuf>>,
}

impl SymbolIndex {
    fn set_roots(&self, roots: Vec<PathBuf>) {
        *self.roots.write().expect("roots lock") = roots;
    }

    fn roots(&self) -> Vec<PathBuf> {
        self.roots.read().expect("roots lock").clone()
    }

    fn index<K>(&self, uri: &Url, text: &str, p: &K)
    where
        K: TypeProvider + ?Sized,
        for<'a> K::View<'a>: TokenKnowledge,
    {
        self.remove(uri);
        let defs = top_level_defs(text, p);
        if defs.is_empty() {
            return;
        }

        let mut names = Vec::with_capacity(defs.len());
        for (name, range, kind) in defs {
            let name = Symbol::new(&name);
            self.defs
                .entry(name.clone())
                .or_default()
                .push(DefSite {
                    uri: uri.clone(),
                    range,
                    kind,
                });
            names.push(name);
        }
        self.files.insert(uri.clone(), names);
    }

    fn remove(&self, uri: &Url) {
        let Some((_, names)) = self.files.remove(uri)
        else {
            return;
        };

        for name in names {
            let empty = self.defs
                .get_mut(&name)
                .is_some_and(|mut sites| {
                    sites.retain(|site| &site.uri != uri);
                    sites.is_empty()
                });
            if empty {
                self.defs.remove(&name);
            }
        }
    }
}

impl DefLookup for SymbolIndex {
    fn find(
        &self,
        name: &str,
        skip: &Url,
    ) -> Vec<Location> {
        self.defs.get(name)
            .into_iter()
            .flat_map(|sites| sites.iter())
            .filter(|site| &site.uri != skip)
            .map(|site| Location {
                uri: site.uri.clone(),
                range: site.range,
            })
            .collect()
    }

    fn has(&self, name: &str) -> bool {
        self.defs.contains_key(name)
    }

    fn kind(
        &self,
        name: &str,
        skip: &Url,
    ) -> Option<TokenKind> {
        self.defs.get(name)?
            .iter()
            .find(|site| &site.uri != skip)
            .map(|site| site.kind)
    }
}
}

#![allow(unused)]
fn main() {
//! A Rust example of a language that uses the tree-sitter-highlight preprocessor.
use dashmap::DashMap;
use lsp_types::{Location, Range, Url};
use std::{path::PathBuf, sync::{Arc, RwLock}};


#[derive(Debug, Clone)]
struct DefSite {
    uri: Url,
    range: Range,
    kind: TokenKind,
}
/// A symbol name.
/// # Examples
/// ```
/// use treesitter_highlight::{Symbol, Symbol::new};
/// let symbol = new("foo");
/// assert_eq!(symbol.name(), "foo");
/// ```
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct Symbol(String);

impl Symbol {
    pub fn new(name: &str) -> Self {
        Self(name.to_string())
    }

    pub fn name(&self) -> &str {
        &self.0
    }
}
trait DefLookup {
    fn find(&self, name: &str, skip: &Url) -> Vec<Location>;
    fn has(&self, name: &str) -> bool; // Comment
    fn kind(&self, name: &str, skip: &Url) -> Option<TokenKind>;
}

impl<T: DefLookup + ?Sized> DefLookup for Arc<T> {
    fn find(
        &self,
        name: &str,
        skip: &Url,
    ) -> Vec<Location> {
        self.as_ref().find(name, skip)
    }

    fn has(&self, name: &str) -> bool {
        self.as_ref().has(name)
    }

    fn kind(
        &self,
        name: &str,
        skip: &Url,
    ) -> Option<TokenKind> {
        self.as_ref().kind(name, skip)
    }
}

/// A symbol index kept in sync with editor changes.
#[derive(Debug, Default)]
struct SymbolIndex {
    defs: DashMap<Symbol, Vec<DefSite>>,
    files: DashMap<Url, Vec<Symbol>>,
    roots: RwLock<Vec<PathBuf>>,
}

impl SymbolIndex {
    fn set_roots(&self, roots: Vec<PathBuf>) {
        *self.roots.write().expect("roots lock") = roots;
    }

    fn roots(&self) -> Vec<PathBuf> {
        self.roots.read().expect("roots lock").clone()
    }

    fn index<K>(&self, uri: &Url, text: &str, p: &K)
    where
        K: TypeProvider + ?Sized,
        for<'a> K::View<'a>: TokenKnowledge,
    {
        self.remove(uri);
        let defs = top_level_defs(text, p);
        if defs.is_empty() {
            return;
        }

        let mut names = Vec::with_capacity(defs.len());
        for (name, range, kind) in defs {
            let name = Symbol::new(&name);
            self.defs
                .entry(name.clone())
                .or_default()
                .push(DefSite {
                    uri: uri.clone(),
                    range,
                    kind,
                });
            names.push(name);
        }
        self.files.insert(uri.clone(), names);
    }

    fn remove(&self, uri: &Url) {
        let Some((_, names)) = self.files.remove(uri)
        else {
            return;
        };

        for name in names {
            let empty = self.defs
                .get_mut(&name)
                .is_some_and(|mut sites| {
                    sites.retain(|site| &site.uri != uri);
                    sites.is_empty()
                });
            if empty {
                self.defs.remove(&name);
            }
        }
    }
}

impl DefLookup for SymbolIndex {
    fn find(
        &self,
        name: &str,
        skip: &Url,
    ) -> Vec<Location> {
        self.defs.get(name)
            .into_iter()
            .flat_map(|sites| sites.iter())
            .filter(|site| &site.uri != skip)
            .map(|site| Location {
                uri: site.uri.clone(),
                range: site.range,
            })
            .collect()
    }

    fn has(&self, name: &str) -> bool {
        self.defs.contains_key(name)
    }

    fn kind(
        &self,
        name: &str,
        skip: &Url,
    ) -> Option<TokenKind> {
        self.defs.get(name)?
            .iter()
            .find(|site| &site.uri != skip)
            .map(|site| site.kind)
    }
}
}

C++

Concepts, ranges, views, variants, smart pointers, lambdas, and move semantics.

#include <algorithm>
#include <concepts>
#include <iostream>
#include <memory>
#include <ranges>
#include <string>
#include <variant>
#include <vector>

struct Text {
    std::string value;
};

struct Number {
    double value;
};

using Cell = std::variant<Text, Number>;

template<typename T>
concept Printable = requires(
    std::ostream& out,
    const T& value
) {
    { out << value } -> std::same_as<std::ostream&>;
};

class Table {
public:
    void add(Cell cell) {
        cells_.push_back(std::move(cell));
    }

    auto numbers() const {
        return cells_
            | std::views::filter([](const Cell& cell) {
                return std::holds_alternative<Number>(
                    cell
                );
            })
            | std::views::transform([](const Cell& cell) {
                return std::get<Number>(cell).value;
            });
    }

private:
    std::vector<Cell> cells_;
};

template<std::ranges::input_range R>
requires std::convertible_to<
    std::ranges::range_value_t<R>,
    double
>
double average(R&& values) {
    double sum = 0.0;
    std::size_t count = 0;
    for (double value : values) {
        sum += value;
        ++count;
    }
    return count ? sum / count : 0.0;
}

int main() {
    auto table = std::make_unique<Table>();
    table->add(Text{"latency"});
    table->add(Number{18.4});
    table->add(Number{21.2});

    std::cout << average(table->numbers()) << '\n';
}

#include <algorithm>
#include <concepts>
#include <iostream>
#include <memory>
#include <ranges>
#include <string>
#include <variant>
#include <vector>

struct Text {
    std::string value;
};

struct Number {
    double value;
};

using Cell = std::variant<Text, Number>;

template<typename T>
concept Printable = requires(
    std::ostream& out,
    const T& value
) {
    { out << value } -> std::same_as<std::ostream&>;
};

class Table {
public:
    void add(Cell cell) {
        cells_.push_back(std::move(cell));
    }

    auto numbers() const {
        return cells_
            | std::views::filter([](const Cell& cell) {
                return std::holds_alternative<Number>(
                    cell
                );
            })
            | std::views::transform([](const Cell& cell) {
                return std::get<Number>(cell).value;
            });
    }

private:
    std::vector<Cell> cells_;
};

template<std::ranges::input_range R>
requires std::convertible_to<
    std::ranges::range_value_t<R>,
    double
>
double average(R&& values) {
    double sum = 0.0;
    std::size_t count = 0;
    for (double value : values) {
        sum += value;
        ++count;
    }
    return count ? sum / count : 0.0;
}

int main() {
    auto table = std::make_unique<Table>();
    table->add(Text{"latency"});
    table->add(Number{18.4});
    table->add(Number{21.2});

    std::cout << average(table->numbers()) << '\n';
}

Macaulay2

Self-initializing types, methods, typed dispatch, hash and cache tables, regular expressions, recursion, loops, control flow, and tree transformations.

Metavar = new SelfInitializingType of TokenTree

metavarNode = method()
metavarNode String := Metavar => name -> Metavar(name, {}, null, null)
metavarNode (String, String) := Metavar => (name, kind) -> Metavar(name, {}, null, kind)

isMetavar = t -> instance(t, Metavar)
metavarName = t -> leftOf t
metavarKind = t -> delimiterOf t

Repetition = new SelfInitializingType of TokenTree

repetitionNode = (quantifier, sep, unit) -> Repetition(quantifier, unit, null, sep)
isRepetition = t -> instance(t, Repetition)
repQuantifier = t -> leftOf t
repSeparator = t -> delimiterOf t
repUnit = t -> contentOf t

Alternation = new SelfInitializingType of TokenTree

alternationNode = branches -> Alternation(null, branches, null, null)
isAlternation = t -> instance(t, Alternation)
altBranches = t -> contentOf t

isSeqNode = t -> not isRepetition t and
    (delimiterOf t === "," or delimiterOf t === ";" or delimiterOf t === statementSeparator)

nodeKind = t -> (
    if isComment t then "Comment"
    else if isMacroInvocation t then "MacroInvocation"
    else if isMetavar t then "Metavar"
    else if isRepetition t then "Repetition"
    else if isAlternation t then "Alternation"
    else if isLeaf t then (
        s := leftOf t;
        if s === null then "Node"
        else if s#0 == "\"" then "String"
        else if match("^[0-9]", s) then "Number"
        else if match("^[A-Za-z]", s) then (if m2Keywords#?s then "Keyword" else "Identifier")
        else "Operator")
    else (
        d := delimiterOf t;
        if d === spaceOperator then "Apply"
        else if d === statementSeparator then "Statements"
        else if d === whitespaceDelimiter then (
            cs := contentOf t;
            if #cs == 0 then "Clause" else capitalize leftOf cs#0)
        else if d === "," or d === ";" then "Sequence"
        else if d === "->" then "Arrow"
        else if instance(d, String) then "Infix"
        else if leftOf t =!= null and rightOf t =!= null then "Bracket"
        else if leftOf t =!= null then "Prefix"
        else if rightOf t =!= null then "Postfix"
        else "Node"))

nodeKindNames = set {"Comment", "MacroInvocation", "Metavar", "Repetition", "Alternation", "String",
    "Number", "Keyword", "Identifier", "Operator", "Apply", "Sequence", "Arrow",
    "Infix", "Bracket", "Prefix", "Postfix", "If", "While", "For", "Try", "New",
    "Statements", "Clause", "Node"}

metavarPlaceholderPrefix = "MetavarHolePlaceholder"
metavarKindPrefix = "MetavarKind"
toPlaceholders = src -> (
    typed := replace(///(?<![A-Za-z0-9'])'([A-Za-z][A-Za-z0-9]*):([A-Za-z][A-Za-z0-9]*)///,
        concatenate(metavarKindPrefix, "$2(", metavarPlaceholderPrefix, "$1)"), src);
    replace(///(?<![A-Za-z0-9'])'([A-Za-z][A-Za-z0-9]*)///, metavarPlaceholderPrefix | "$1", typed))

repCallNames = new HashTable from {"+" => "RepPlus", "*" => "RepStar", "|" => "Alt"}
isIdentChar = c -> match("[A-Za-z0-9']", c)
scanReps = src -> (
    n := #src;
    at := i -> if i >= 0 and i < n then substring(i, 1, src) else "";
    stack := {};
    spans := {};
    for i to n - 1 do (
        c := at i;
        if c == "{" then (
            isFormOpen := i >= 1 and at(i - 1) == "'" and (i < 2 or not isIdentChar at(i - 2));
            stack = append(stack, (i, isFormOpen)))
        else if c == "}" then (
            if #stack == 0 then error "scanReps: unbalanced }";
            top := last stack;
            stack = drop(stack, -1);
            if top#1 then (
                form := if at(i + 1) == "+" or at(i + 1) == "*" then at(i + 1) else "|";
                spans = append(spans, (top#0 - 1, i, form)))));
    if #spans == 0 then return src;
    opens := hashTable apply(spans, s -> (s#0, repCallNames#(s#2) | "("));
    closes := hashTable apply(spans, s -> (s#1, if s#2 === "|" then 1 else 2));
    out := "";
    j := 0;
    while j < n do (
        if opens#?j then (out |= opens#j;
            j += 2)
        else if closes#?j then (out |= ")";
            j += closes#j)
        else (out |= at j;
            j += 1));
    out)

quantifierOf = t -> (
    if delimiterOf t === spaceOperator and #contentOf t == 2 and isLeaf (contentOf t)#0
    then (n := leftOf (contentOf t)#0;
        if n === "RepPlus" then "+" else if n === "RepStar" then "*"))

isNullElement = t -> isLeaf t and leftOf t === "null"
unitOf = t -> (
    inner := (contentOf (contentOf t)#1)#0;
    sep := if isSeqNode inner then delimiterOf inner else ",";
    elems := if isSeqNode inner then contentOf inner else {inner};
    while #elems > 0 and isNullElement last elems do elems = drop(elems, -1);
    (sep, elems))

altCallName = "Alt"
altInnerOf = t -> (
    if delimiterOf t === spaceOperator and #contentOf t == 2 and isLeaf (contentOf t)#0
    and leftOf (contentOf t)#0 === altCallName
    then (inner := contentOf (contentOf t)#1;
        if #inner == 0 then error "empty '{ | } alternation";
        inner#0))

altBranchesOf = t -> (
    if delimiterOf t === "|" and #contentOf t == 2
    then join(altBranchesOf (contentOf t)#0, altBranchesOf (contentOf t)#1)
    else {t})

typedKindOf = t -> (
    if delimiterOf t === spaceOperator and #contentOf t == 2 and isLeaf (contentOf t)#0
    and match("^" | metavarKindPrefix, leftOf (contentOf t)#0)
    then substring(#metavarKindPrefix, leftOf (contentOf t)#0))

markNodes = t -> (
    if isLeaf t then (
        if leftOf t =!= null and match("^" | metavarPlaceholderPrefix, leftOf t)
        then metavarNode substring(#metavarPlaceholderPrefix, leftOf t) else t)
    else if typedKindOf t =!= null then (
        kind := typedKindOf t;
        if not nodeKindNames#?kind then error("unknown node kind in pattern: '" | kind);
        hole := leftOf (contentOf (contentOf t)#1)#0;
        metavarNode(substring(#metavarPlaceholderPrefix, hole), kind))
    else if quantifierOf t =!= null then (
        (sep, elems) := unitOf t;
        repetitionNode(quantifierOf t, sep, apply(elems, markNodes)))
    else if altInnerOf t =!= null then
        alternationNode apply(altBranchesOf altInnerOf t, markNodes)
    else (setContent(t, apply(contentOf t, markNodes));
        t))

templateCache = new CacheTable
parseTemplate = src ->
templateCache#src ??= markNodes parseMacroTree toPlaceholders scanReps src

metavarNamesIn = t -> (
    if isMetavar t then {metavarName t}
    else flatten apply(contentOf t, metavarNamesIn))

treeEquals = (a, b) -> (
    leftOf a === leftOf b and rightOf a === rightOf b and delimiterOf a === delimiterOf b
    and #contentOf a == #contentOf b
    and all(#contentOf a, i -> treeEquals((contentOf a)#i, (contentOf b)#i)))

matchRepetition = (rep, ielems, b) -> (
    unit := repUnit rep;
    u := #unit;
    if u == 0 then error "empty '{ } repetition unit";
    scan(metavarNamesIn rep, nm -> if b#?nm and not instance(b#nm, List) then
            error("metavariable '" | nm | " is bound both outside and inside a repetition"));
    if #ielems % u != 0 then return false;
    nChunks := #ielems // u;
    if repQuantifier rep === "+" and nChunks == 0 then return false;
    ok := all(nChunks, ci -> (
        tb := new MutableHashTable;
        chunkOK := all(u, j -> matchInto(unit#j, ielems#(ci * u + j), tb));
        if chunkOK then scan(keys tb, nm -> b#nm = append(b#nm ?? {}, tb#nm));
        chunkOK));
    if ok and nChunks == 0 then scan(metavarNamesIn rep, nm -> b#nm ??= {});
    ok)

matchElems = (pelems, ielems, b) -> (
    reps := positions(pelems, isRepetition);
    if #reps == 0 then #pelems == #ielems and all(#pelems, i -> matchInto(pelems#i, ielems#i, b))
    else if #reps > 1 then error "a pattern sequence may hold at most one '{ } repetition"
    else (
        r := first reps;
        before := take(pelems, r);
        after := drop(pelems, r + 1);
        if #ielems < #before + #after then return false;
        nRep := #ielems - #before - #after;
        all(#before, i -> matchInto(before#i, ielems#i, b))
        and all(#after, i -> matchInto(after#i, ielems#(#before + nRep + i), b))
        and matchRepetition(pelems#r, take(drop(ielems, #before), nRep), b)))

matchInto = (pat, inp, b) -> (
    if isMetavar pat then (
        if metavarKind pat =!= null and nodeKind inp =!= metavarKind pat then false
        else (
            name := metavarName pat;
            if b#?name then treeEquals(b#name, inp)
            else (b#name = inp;
                true))
    )
    else if isAlternation pat then (
        matched := false;
        for branch in altBranches pat when not matched do (
            tb := new MutableHashTable;
            if matchInto(branch, inp, tb) and all(keys tb, k -> not b#?k or treeEquals(b#k, tb#k))
            then (scan(keys tb, k -> b#k = tb#k);
                matched = true));
        matched
    )
    else if isRepetition pat then
        matchRepetition(pat, if isSeqNode inp then contentOf inp else {inp}, b)
    else if isSeqNode pat and any(contentOf pat, isRepetition) then (
        if isSeqNode inp and delimiterOf pat === delimiterOf inp then matchElems(contentOf pat,
            contentOf inp, b)
        else if isSeqNode inp then false
        else matchElems(contentOf pat, {inp}, b)
    )
    else if #contentOf pat == 1 and isRepetition first contentOf pat
    and leftOf pat === leftOf inp and rightOf pat === rightOf inp
    and delimiterOf pat === delimiterOf inp then
        matchRepetition(first contentOf pat,
        flatten apply(contentOf inp, ic -> if isSeqNode ic then contentOf ic else {ic}), b)
    else if leftOf pat =!= leftOf inp or rightOf pat =!= rightOf inp
    or delimiterOf pat =!= delimiterOf inp
    or #contentOf pat =!= #contentOf inp then false
    else (
        cs := contentOf pat;
        ds := contentOf inp;
        all(#cs, i -> matchInto(cs#i, ds#i, b))
    ))

matchPattern = (pat, inp) -> (
    b := new MutableHashTable;
    if matchInto(pat, inp, b) then new HashTable from b)

cloneTree = t -> (class t)(leftOf t, apply(contentOf t, cloneTree), rightOf t, delimiterOf t)

expandRepetition = (rep, b) -> (
    unit := repUnit rep;
    names := select(metavarNamesIn rep, nm -> b#?nm);
    lengths := unique apply(names, nm -> #(b#nm));
    if #lengths > 1 then error "template repetition metavariables have differing lengths";
    nReps := if #names == 0 then 0 else first lengths;
    flatten apply(nReps, i -> (
        perRep := hashTable apply(names, nm -> (nm, (b#nm)#i));
        apply(unit, u -> instantiate(u, perRep)))))

instantiate = (tmpl, b) -> (
    if isAlternation tmpl then
        error "alternation '{ a | b } is a pattern-only construct, not valid in a template";
    if isMetavar tmpl then (
        name := metavarName tmpl;
        if not b#?name
        then error("template metavariable '" | name | " is unbound");
        cloneTree b#name
    )
    else if any(contentOf tmpl, isRepetition) then (
        if isSeqNode tmpl then
            TokenTree(leftOf tmpl,
            flatten apply(contentOf tmpl, c -> if isRepetition c then expandRepetition(c, b) else {
                instantiate(c, b)}),
            rightOf tmpl, delimiterOf tmpl)
        else (
            if #contentOf tmpl != 1 then
                error "a repetition '{ }+ in a template must be the only content of a sequence or bracket";
            rep := first contentOf tmpl;
            inner := delimited(repSeparator rep, expandRepetition(rep, b));
            TokenTree(leftOf tmpl, {inner}, rightOf tmpl, delimiterOf tmpl))
    )
    else (class tmpl)(leftOf tmpl, apply(contentOf tmpl, c -> instantiate(c, b)), rightOf tmpl,
        delimiterOf tmpl))

quote = method(Dispatch => Thing)
quote String := TokenTree => src -> instantiate(parseTemplate src, new HashTable)
quote Sequence := TokenTree => s -> (
    rest := drop(s, 1);
    binding := if #rest == 1 and instance(first rest, HashTable) then first rest
        else hashTable apply(rest, o -> (toString o#0, o#1));
    instantiate(parseTemplate first s, binding))

matchesIn = method()
matchesIn (TokenTree, TokenTree) := List => (pat, tree) -> (
    below := flatten apply(contentOf tree, c -> matchesIn(pat, c));
    here := matchPattern(pat, tree);
    if here =!= null then prepend((tree, here), below) else below)
patternCell = src -> (
    p := parseTemplate src;
    if delimiterOf p === statementSeparator and #contentOf p == 1 then (contentOf p)#0 else p)
matchesIn (String, TokenTree) := List => (patSrc, tree) -> matchesIn(patternCell patSrc, tree)

expandRules = (name, rules, inp) -> (
    for r in rules do (
        (pat, tmpl) := r;
        b := matchPattern(pat, inp);
        if b =!= null then
            return instantiate(tmpl, b)
    );
    error(name | ": no rule matched the input"))

declMacro = method()

declMacro (String, List) := Macro => (name, rules) -> (
    scan(rules, r -> if not ((instance(r, Sequence) or instance(r, List)) and #r == 2) then
            error(name | ": each rule must be a (pattern, template) pair, got " | toString r));
    parsed := apply(rules, r -> (parseTemplate r#0, parseTemplate r#1));
    installMacro(name, ts -> expandRules(name, parsed, focus ts)))

declMacro (String, String, String) := Macro => (name, p, t) -> declMacro(name, {(p, t)})

Haskell

Algebraic data types, type classes, constrained polymorphism, higher-order folds, pattern matching, comprehensions, and concurrent traversal.

{-# LANGUAGE DeriveFunctor #-}
{-# LANGUAGE LambdaCase #-}

module Pipeline where

import Control.Concurrent.Async (mapConcurrently)
import Data.Map.Strict (Map)
import qualified Data.Map.Strict as Map

data Result a
  = Success a
  | Failure String
  deriving (Eq, Show, Functor)

data Job a = Job
  { jobName :: String
  , jobInput :: a
  } deriving (Eq, Show, Functor)

class Runnable task where
  run :: task a -> IO (Result a)

instance Runnable Job where
  run job
    | null (jobName job) =
        pure (Failure "missing name")
    | otherwise =
        pure (Success (jobInput job))

partitionResults
  :: [Result a]
  -> ([String], [a])
partitionResults = foldr step ([], [])
  where
    step result (errors, values) =
      case result of
        Failure message ->
          (message : errors, values)
        Success value ->
          (errors, value : values)

runAll
  :: Runnable task
  => [task a]
  -> IO (Map String a)
runAll jobs = do
  results <- mapConcurrently run jobs
  let (_, values) = partitionResults results
      names = ["job-" <> show n | n <- [1 :: Int ..]]
  pure (Map.fromList (zip names values))

describe :: Result a -> String
describe = \case
  Success _ -> "completed"
  Failure message -> "failed: " <> message

{-# LANGUAGE DeriveFunctor #-}
{-# LANGUAGE LambdaCase #-}

module Pipeline where

import Control.Concurrent.Async (mapConcurrently)
import Data.Map.Strict (Map)
import qualified Data.Map.Strict as Map

data Result a
  = Success a
  | Failure String
  deriving (Eq, Show, Functor)

data Job a = Job
  { jobName :: String
  , jobInput :: a
  } deriving (Eq, Show, Functor)

class Runnable task where
  run :: task a -> IO (Result a)

instance Runnable Job where
  run job
    | null (jobName job) =
        pure (Failure "missing name")
    | otherwise =
        pure (Success (jobInput job))

partitionResults
  :: [Result a]
  -> ([String], [a])
partitionResults = foldr step ([], [])
  where
    step result (errors, values) =
      case result of
        Failure message ->
          (message : errors, values)
        Success value ->
          (errors, value : values)

runAll
  :: Runnable task
  => [task a]
  -> IO (Map String a)
runAll jobs = do
  results <- mapConcurrently run jobs
  let (_, values) = partitionResults results
      names = ["job-" <> show n | n <- [1 :: Int ..]]
  pure (Map.fromList (zip names values))

describe :: Result a -> String
describe = \case
  Success _ -> "completed"
  Failure message -> "failed: " <> message

Lua

Metatables, method syntax, closures, coroutine-based iteration, higher-order functions, table operations, and protected calls.

local Stream = {}
Stream.__index = Stream

function Stream.new(source)
  return setmetatable({
    source = source,
    steps = {},
  }, Stream)
end

function Stream:map(fn)
  self.steps[#self.steps + 1] = function(value)
    return true, fn(value)
  end
  return self
end

function Stream:filter(predicate)
  self.steps[#self.steps + 1] = function(value)
    return predicate(value), value
  end
  return self
end

function Stream:iter()
  return coroutine.wrap(function()
    for value in self.source do
      local keep = true
      local current = value
      for _, step in ipairs(self.steps) do
        keep, current = step(current)
        if not keep then break end
      end
      if keep then coroutine.yield(current) end
    end
  end)
end

local function values(items)
  local index = 0
  return function()
    index = index + 1
    return items[index]
  end
end

local stream = Stream.new(values({ 1, 2, 3, 4, 5 }))
  :filter(function(n) return n % 2 == 1 end)
  :map(function(n) return n * n end)

local ok, err = pcall(function()
  for value in stream:iter() do
    print(("square: %d"):format(value))
  end
end)

if not ok then
  io.stderr:write(err, "\n")
end

local Stream = {}
Stream.__index = Stream

function Stream.new(source)
  return setmetatable({
    source = source,
    steps = {},
  }, Stream)
end

function Stream:map(fn)
  self.steps[#self.steps + 1] = function(value)
    return true, fn(value)
  end
  return self
end

function Stream:filter(predicate)
  self.steps[#self.steps + 1] = function(value)
    return predicate(value), value
  end
  return self
end

function Stream:iter()
  return coroutine.wrap(function()
    for value in self.source do
      local keep = true
      local current = value
      for _, step in ipairs(self.steps) do
        keep, current = step(current)
        if not keep then break end
      end
      if keep then coroutine.yield(current) end
    end
  end)
end

local function values(items)
  local index = 0
  return function()
    index = index + 1
    return items[index]
  end
end

local stream = Stream.new(values({ 1, 2, 3, 4, 5 }))
  :filter(function(n) return n % 2 == 1 end)
  :map(function(n) return n * n end)

local ok, err = pcall(function()
  for value in stream:iter() do
    print(("square: %d"):format(value))
  end
end)

if not ok then
  io.stderr:write(err, "\n")
end

Python

Data classes, protocols, generics, async iteration, task groups, comprehensions, enums, and structural pattern matching.

from collections.abc import AsyncIterator
from dataclasses import dataclass, field
from enum import StrEnum, auto
from typing import Protocol, TypeVar
import asyncio

T = TypeVar("T")


class State(StrEnum):
    QUEUED = auto()
    RUNNING = auto()
    DONE = auto()
    FAILED = auto()


@dataclass(slots=True)
class Job:
    name: str
    command: tuple[str, ...]
    state: State = State.QUEUED
    output: list[str] = field(default_factory=list)


class Store(Protocol[T]):
    async def save(self, key: str, value: T) -> None: ...
    async def load(self, key: str) -> T | None: ...


async def run(job: Job) -> AsyncIterator[str]:
    job.state = State.RUNNING
    proc = await asyncio.create_subprocess_exec(
        *job.command,
        stdout=asyncio.subprocess.PIPE,
    )
    assert proc.stdout is not None

    async for raw in proc.stdout:
        line = raw.decode().rstrip()
        job.output.append(line)
        yield line

    code = await proc.wait()
    job.state = State.DONE if code == 0 else State.FAILED


async def supervise(
    jobs: list[Job],
    store: Store[Job],
) -> dict[State, int]:
    async with asyncio.TaskGroup() as group:
        for job in jobs:
            group.create_task(store.save(job.name, job))

    counts = {state: 0 for state in State}
    for job in jobs:
        match job.state:
            case State.FAILED if job.output:
                print(f"{job.name}: {job.output[-1]}")
            case State.DONE:
                print(f"{job.name}: complete")
            case _:
                pass
        counts[job.state] += 1
    return counts

from collections.abc import AsyncIterator
from dataclasses import dataclass, field
from enum import StrEnum, auto
from typing import Protocol, TypeVar
import asyncio

T = TypeVar("T")


class State(StrEnum):
    QUEUED = auto()
    RUNNING = auto()
    DONE = auto()
    FAILED = auto()


@dataclass(slots=True)
class Job:
    name: str
    command: tuple[str, ...]
    state: State = State.QUEUED
    output: list[str] = field(default_factory=list)


class Store(Protocol[T]):
    async def save(self, key: str, value: T) -> None: ...
    async def load(self, key: str) -> T | None: ...


async def run(job: Job) -> AsyncIterator[str]:
    job.state = State.RUNNING
    proc = await asyncio.create_subprocess_exec(
        *job.command,
        stdout=asyncio.subprocess.PIPE,
    )
    assert proc.stdout is not None

    async for raw in proc.stdout:
        line = raw.decode().rstrip()
        job.output.append(line)
        yield line

    code = await proc.wait()
    job.state = State.DONE if code == 0 else State.FAILED


async def supervise(
    jobs: list[Job],
    store: Store[Job],
) -> dict[State, int]:
    async with asyncio.TaskGroup() as group:
        for job in jobs:
            group.create_task(store.save(job.name, job))

    counts = {state: 0 for state in State}
    for job in jobs:
        match job.state:
            case State.FAILED if job.output:
                print(f"{job.name}: {job.output[-1]}")
            case State.DONE:
                print(f"{job.name}: complete")
            case _:
                pass
        counts[job.state] += 1
    return counts

C

Enums, tagged unions, slices, function pointers, dynamic storage, designated initializers, and explicit cleanup.

#include <stdbool.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>

typedef enum {
    EVENT_DATA,
    EVENT_ERROR,
    EVENT_CLOSED,
} EventKind;

typedef struct {
    const char *data;
    size_t length;
} Slice;

typedef struct {
    EventKind kind;
    union {
        Slice data;
        int error_code;
    };
} Event;

typedef bool (*EventHandler)(
    const Event *event,
    void *context
);

typedef struct {
    Event *items;
    size_t length;
    size_t capacity;
} EventQueue;

static bool queue_push(
    EventQueue *queue,
    Event event
) {
    if (queue->length == queue->capacity) {
        size_t capacity =
            queue->capacity ? queue->capacity * 2 : 8;
        Event *items = realloc(
            queue->items,
            capacity * sizeof(*items)
        );
        if (items == NULL) {
            return false;
        }
        queue->items = items;
        queue->capacity = capacity;
    }

    queue->items[queue->length++] = event;
    return true;
}

static void queue_drain(
    EventQueue *queue,
    EventHandler handle,
    void *context
) {
    for (size_t i = 0; i < queue->length; ++i) {
        if (!handle(&queue->items[i], context)) {
            break;
        }
    }
    queue->length = 0;
}

static bool print_event(
    const Event *event,
    void *context
) {
    FILE *output = context;
    switch (event->kind) {
    case EVENT_DATA:
        fprintf(
            output,
            "%.*s\n",
            (int)event->data.length,
            event->data.data
        );
        return true;
    case EVENT_ERROR:
        fprintf(output, "error %d\n", event->error_code);
        return false;
    case EVENT_CLOSED:
        return false;
    }
    return false;
}

int main(void) {
    EventQueue queue = {0};
    queue_push(&queue, (Event){
        .kind = EVENT_DATA,
        .data = {"ready", 5},
    });
    queue_drain(&queue, print_event, stdout);
    free(queue.items);
}

#include <stdbool.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>

typedef enum {
    EVENT_DATA,
    EVENT_ERROR,
    EVENT_CLOSED,
} EventKind;

typedef struct {
    const char *data;
    size_t length;
} Slice;

typedef struct {
    EventKind kind;
    union {
        Slice data;
        int error_code;
    };
} Event;

typedef bool (*EventHandler)(
    const Event *event,
    void *context
);

typedef struct {
    Event *items;
    size_t length;
    size_t capacity;
} EventQueue;

static bool queue_push(
    EventQueue *queue,
    Event event
) {
    if (queue->length == queue->capacity) {
        size_t capacity =
            queue->capacity ? queue->capacity * 2 : 8;
        Event *items = realloc(
            queue->items,
            capacity * sizeof(*items)
        );
        if (items == NULL) {
            return false;
        }
        queue->items = items;
        queue->capacity = capacity;
    }

    queue->items[queue->length++] = event;
    return true;
}

static void queue_drain(
    EventQueue *queue,
    EventHandler handle,
    void *context
) {
    for (size_t i = 0; i < queue->length; ++i) {
        if (!handle(&queue->items[i], context)) {
            break;
        }
    }
    queue->length = 0;
}

static bool print_event(
    const Event *event,
    void *context
) {
    FILE *output = context;
    switch (event->kind) {
    case EVENT_DATA:
        fprintf(
            output,
            "%.*s\n",
            (int)event->data.length,
            event->data.data
        );
        return true;
    case EVENT_ERROR:
        fprintf(output, "error %d\n", event->error_code);
        return false;
    case EVENT_CLOSED:
        return false;
    }
    return false;
}

int main(void) {
    EventQueue queue = {0};
    queue_push(&queue, (Event){
        .kind = EVENT_DATA,
        .data = {"ready", 5},
    });
    queue_drain(&queue, print_event, stdout);
    free(queue.items);
}

TypeScript

Discriminated unions, generic interfaces, private fields, async generators, type narrowing, and satisfies.

type Loading = { state: "loading"; id: string };
type Ready<T> = { state: "ready"; value: T };
type Failed = { state: "failed"; error: Error };
type Result<T> = Loading | Ready<T> | Failed;

interface Cache<T extends { id: string }> {
  get(id: string): Promise<T | undefined>;
  put(value: T): Promise<void>;
}

class MemoryCache<T extends { id: string }>
  implements Cache<T> {
  readonly #items = new Map<string, T>();

  async get(id: string): Promise<T | undefined> {
    return this.#items.get(id);
  }

  async put(value: T): Promise<void> {
    this.#items.set(value.id, value);
  }
}

async function* pages<T>(
  url: URL,
  decode: (raw: unknown) => T[],
): AsyncGenerator<T> {
  let next: URL | undefined = url;

  while (next) {
    const response = await fetch(next);
    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`);
    }

    const body: unknown = await response.json();
    for (const item of decode(body)) {
      yield item;
    }

    const link = response.headers.get("x-next");
    next = link ? new URL(link, next) : undefined;
  }
}

function describe<T>(result: Result<T>): string {
  switch (result.state) {
    case "loading":
      return `loading ${result.id}`;
    case "ready":
      return JSON.stringify(result.value);
    case "failed":
      return result.error.message;
  }
}

const config = {
  retries: 3,
  mode: "eager",
} as const satisfies Record<string, unknown>;