mmo-ref/mudlet
1
0
Fork 0
mirror of https://github.com/Mudlet/Mudlet synced 2026-08-13 18:26:27 -04:00
Code Issues Projects Releases Packages Wiki Activity

infrastructure: add performance baseline benchmark for the text and trigger pipeline (#9509)

Browse source 
#### Brief overview of PR changes/additions

- Adds `test/functional_tests/PipelineBenchmark.cpp` - a headless,
deterministic, report-only benchmark. It feeds a fixed 25,000-line
corpus (plain text, ANSI SGR colour, UTF-8, and long wrapping-heavy
prose adopted from the Stressinator display package) through the
production `cTelnet -> TBuffer -> TConsole -> TriggerUnit` path via
`loopbackTest()`, and prints `METRIC` lines: text-pipeline throughput,
trigger-engine throughput with a realistic ~34-trigger set, derived
trigger overhead, and peak RSS.
- Adds `test/compare-perf-baseline.py` - the primary workflow. Given an
older and a newer build run on the **same machine**, it parses their
`METRIC` output, prints per-metric deltas, and exits non-zero on a
PASS/FAIL against the 10% gate.
- `test/functional_tests/CMakeLists.txt` builds the benchmark **always**
but keeps it **out of the default ctest suite**; it is report-only and
slow, so it should not run on every CI pipeline. Opt in with
`-DREGISTER_PERF_BENCHMARK=ON` to also register it with ctest.
- Adds `docs/libmudlet-perf-baseline.md` documenting the before/after
workflow. **No canonical, machine-specific numbers are committed** - the
figures in the doc are explicitly illustrative.

#### Motivation for adding to Mudlet

The libmudlet refactor's "no more than 10% throughput loss" gate is
unenforceable without a reproducible way to measure it. Absolute numbers
are meaningless across machines, so this provides a deterministic
harness plus a same-machine before/after comparison tool that turns the
gate into a mechanical PASS/FAIL.

#### Other info (issues closed, discussion etc)

Part of the libmudlet refactor (#8681, #9011) - referenced, not closed.

- **Report-only**: it makes no timing assertions (absolute speed varies
wildly between machines and CI runners), but it does assert the pipeline
genuinely processed data - console buffer fill, every trigger
compiled/registered, and an untimed sentinel trigger firing - so a
silently-disconnected pipeline fails instead of reporting inflated
numbers.
- Each phase feeds the corpus 6 times and reports the **fastest pass**:
the least-disturbed pass isolates intrinsic speed from transient CPU
contention, keeping run-to-run spread ~2% even on a loaded machine.
- **Companion, not a replacement, for the live-GUI path.**
`PipelineBenchmark` runs offscreen and covers the telnet -> buffer ->
trigger core (the piece the refactor moves). The **Stressinator display
benchmark** covers the on-screen render/echo path on a live build; its
wrapping-heavy prose has been adopted into this corpus. Between them
they cover bytes-off-the-socket to pixels-on-screen.

Assisted-by: Claude:claude-fable-5

**Test case:**

Primary workflow - build an older and a newer tree on the same machine,
then compare:

```
flock /tmp/mudlet-functional-tests.lock \
  test/compare-perf-baseline.py --run \
    ../mudlet-before/build/test/functional_tests/PipelineBenchmark \
    build/test/functional_tests/PipelineBenchmark
```

Or run the benchmark once directly (it is built even without the ctest
opt-in):

```
QT_QPA_PLATFORM=offscreen ASAN_OPTIONS=detect_leaks=0 \
  ./build/test/functional_tests/PipelineBenchmark
```

Illustrative output (absolute values vary per machine; nothing is
asserted on timing):

```
METRIC text_corpus_lines 25000
METRIC text_corpus_bytes 1436934
METRIC text_lines_per_sec 4281.46
METRIC text_mb_per_sec 0.41
METRIC text_best_pass_ms 5839.12
METRIC trigger_count 34
METRIC trigger_lines_per_sec 3323.25
METRIC trigger_mb_per_sec 0.32
METRIC trigger_best_pass_ms 7523.40
METRIC trigger_overhead_ms 1683.64
METRIC peak_rss_kb 1402384
```

To drive it through ctest instead, configure with
`-DREGISTER_PERF_BENCHMARK=ON`, then `ctest -R PipelineBenchmark -V`.
This commit is contained in:
Vadim Peretokin 2026-07-27 22:01:25 +02:00 • committed by GitHub
parent ed6ffdd898
commit 7d67d4bfb9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 1002 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

218
docs/libmudlet-perf-baseline.md Normal file
View file

@ -0,0 +1,218 @@
# libmudlet performance baseline
The libmudlet refactor (extracting Qt Widgets from `mudlet_core`, issue #9011)
carries a **"no more than 10% throughput loss"** gate. Absolute benchmark
numbers are specific to the machine that produced them, so **none are committed
here as a target**. What matters is running the benchmark on an *older* and a
*newer* Mudlet **on the same machine** and comparing the two. This document
explains how to do that.
## The harness
The harness is the `PipelineBenchmark` functional test
(`test/functional_tests/PipelineBenchmark.cpp`). It drives a real Mudlet profile
with a fixed, deterministically-generated corpus (mixed plain text, ANSI SGR
colour, UTF-8 and long wrapping-heavy prose) through the production
`cTelnet::processSocketData -> TBuffer::translateToPlainText -> TConsole ->
TriggerUnit` path via `cTelnet::loopbackTest()` - the same code an online
session runs (that path internally calls `TMainConsole::printOnDisplay()`, so the
Lua `feedTriggers()` entry point is covered too) - and measures:
- **text pipeline throughput** - no triggers active (`text_*` metrics)
- **trigger engine throughput** - the same corpus with a realistic ~34-trigger
set active covering the plain substring, Perl regex with capture groups,
begin-of-line substring, ANSI colour and multiline matcher kinds
(`trigger_*` metrics). Lua-code matchers are deliberately excluded to keep
Lua execution out of the timed path, and prompt triggers need a GA signal a
loopback feed cannot produce.
- **peak resident set size** for the whole process, from `/proc/self/status`
`VmHWM` on Linux (`peak_rss_kb`)
It is **report-only**: it makes no timing assertions (absolute speed varies
wildly between machines and CI runners) and always passes as long as the
pipeline actually processed data - which is genuinely asserted: each phase
verifies the console buffer filled to its scrollback cap, the trigger phase
verifies every trigger compiled and registered, and an untimed sentinel trigger
proves the trigger engine consumes what the loopback path feeds. A
silently-disconnected pipeline fails the run instead of reporting
impressive-looking garbage. Each phase feeds the corpus several times and
reports the **fastest single pass** - the least-disturbed pass isolates the
code's intrinsic speed from transient CPU contention, which is exactly what a
before/after comparison needs. That makes the numbers stable (~2% run-to-run)
even on a shared/CI box running other builds alongside it.
## How to run
`PipelineBenchmark` is **built as part of the functional tests but deliberately
not registered with ctest by default** - it is report-only and feeds a huge
corpus many times, so it would only burn minutes on every CI run. Run it
directly instead (which is exactly what the compare script does):
```bash
cd <build-dir>
# single run, human-readable + METRIC lines:
QT_QPA_PLATFORM=offscreen ASAN_OPTIONS=detect_leaks=0 \
./test/functional_tests/PipelineBenchmark
```
If you want to drive it through ctest (e.g. `ctest -R PipelineBenchmark -V`,
which reuses the offscreen/ASAN env from CMake), configure the build with the
opt-in first:
```bash
cmake -S . -B <build-dir> -DREGISTER_PERF_BENCHMARK=ON
ctest --test-dir <build-dir> -R PipelineBenchmark -V
```
On a machine with other functional-test runs (e.g. parallel worktrees) wrap the
command in `flock /tmp/mudlet-functional-tests.lock ...` so stub ports and the
shared config directory do not collide.
Results are printed one per line as `METRIC <name> <value>`, so runs can be
diffed mechanically (the values below are illustrative, not a target):
```
METRIC build_asan 1
METRIC text_lines_per_sec 4281.46
METRIC text_mb_per_sec 0.41
METRIC trigger_lines_per_sec 3323.25
METRIC trigger_overhead_ms 1683.64
METRIC peak_rss_kb 1402384
...
```
## The before/after workflow (the 10% gate)
The gate is a **relative, same-machine** comparison. Never compare numbers taken
on different hardware, or from an ASan build against a release build - only ever
*old vs new on one machine, built the same way*.
1. **Build the "before" tree.** Check out the branch point (before the change
under test), configure and build the functional tests, and keep that build
directory.
```bash
git worktree add ../mudlet-before <base-commit>
cmake -S ../mudlet-before -B ../mudlet-before/build -G Ninja
cmake --build ../mudlet-before/build --target PipelineBenchmark
```
2. **Build the "after" tree** the same way from your changed branch (e.g. the
current `build/`).
3. **Run and compare** with the helper, which runs both binaries and prints the
per-metric delta with a PASS/FAIL against the threshold (default 10%):
```bash
flock /tmp/mudlet-functional-tests.lock \
test/compare-perf-baseline.py --run \
../mudlet-before/build/test/functional_tests/PipelineBenchmark \
build/test/functional_tests/PipelineBenchmark
```
Or capture each run to a file and compare the files (handy when the two
builds live on different checkouts or you want to keep a record):
```bash
QT_QPA_PLATFORM=offscreen ASAN_OPTIONS=detect_leaks=0 \
../mudlet-before/build/test/functional_tests/PipelineBenchmark > before.txt
QT_QPA_PLATFORM=offscreen ASAN_OPTIONS=detect_leaks=0 \
build/test/functional_tests/PipelineBenchmark > after.txt
test/compare-perf-baseline.py before.txt after.txt
```
`compare-perf-baseline.py` gates on `text_lines_per_sec` and
`trigger_lines_per_sec` by default (the two throughput numbers); every other
metric is reported for context. It exits non-zero if any gated metric regressed
by more than the threshold, so it drops straight into a script or CI step. Tune
it with `--threshold 0.10` and `--gate metric,metric,...`. A `--threshold` of 1
or more is read as a percentage (e.g. `--threshold 10` means 10%), with a note
on stderr; `--threshold 0` or a negative value is rejected.
Because it is the arbiter of the gate, the script refuses (exit code 2) rather
than silently passing whenever it cannot trust the comparison:
- an invariant (`text_corpus_lines`, `text_corpus_bytes`, `trigger_count`,
`build_asan`) is missing from either run, or differs between them - the two
runs used different corpora, trigger sets or build flavours. `build_asan`
specifically stops an ASan build being compared against a release build.
- a **gated** metric is missing from either run, or its "before" value is not
positive (a valid throughput/time baseline must be greater than zero).
- a `--gate` name matches no metric in either run (usually a typo).
- any `METRIC` line fails to parse fully - a non-numeric, NaN/Inf or
comma-decimal value, or a duplicate metric name. Such a line is never dropped
silently, because a vanished gated metric would otherwise let the gate pass.
**Gating on `trigger_overhead_ms` (opt-in).** Trigger throughput includes the
text-pipeline cost, which dilutes a matcher-only regression roughly 4x.
`trigger_overhead_ms` (trigger best pass minus text best pass - valid because
both phases feed identical bytes) isolates the matching engine itself. It is
**not gated by default**, though, because it is the difference of two
independently-noisy best passes: their noise adds, so its worst-case run-to-run
spread (~16%) is wider than the 10% gate and it would fire on noise alone. When a
change specifically targets trigger matching, gate on it explicitly and confirm
the movement is real - `--gate text_lines_per_sec,trigger_lines_per_sec,trigger_overhead_ms`,
ideally over a couple of runs or with a slightly relaxed threshold.
## Companion: Stressinator (live GUI display path)
`PipelineBenchmark` deliberately stops at the core pipeline: it runs offscreen
and never paints a widget, so it does not measure the on-screen rendering and
echo path. That path needs a live window and is covered by the **Stressinator
display benchmark** (`src/mudlet-lua/lua/stressinator/StressinatorDisplayBench.xml`),
pre-installed into the `mudlet.org` self-test profile.
- Interactively, in a running profile, type `stresstest 100000` to feed that many
lines of prose through `feedTriggers()` and print the average per-line time.
- In CI it runs automatically: `.github/workflows/performance-analysis.yml`
launches Mudlet on a fixed self-hosted machine with
`AUTORUN_DISPLAY_BENCHMARK=true` and appends the per-line result to a
spreadsheet, tracking display throughput over time.
The two are complementary. Use `PipelineBenchmark` for a deterministic, headless,
CI-able check of the telnet -> buffer -> trigger core (the piece the libmudlet
refactor moves), and run Stressinator on a live build when you need to confirm
the rendering/echo path did not regress. Between them they cover the pipeline
from bytes-off-the-socket to pixels-on-screen.
## Illustrative example output (NOT a target)
The table below is **an example of one run on one machine, kept only to show the
shape and rough ratios of the output**. Do not treat any figure here as a target
or a committed baseline - capture your own "before" on the machine you are
testing on and compare against that.
| Metric | Example value |
| --- | --- |
| `text_lines_per_sec` | ~4,270 |
| `text_mb_per_sec` | ~0.41 |
| `text_best_pass_ms` | ~5,850 (25,000 lines/pass) |
| `trigger_lines_per_sec` | ~3,320 |
| `trigger_mb_per_sec` | ~0.32 |
| `trigger_best_pass_ms` | ~7,530 |
| `trigger_overhead_ms` | ~1,670 |
| `trigger_count` | 34 |
| `peak_rss_kb` | ~1,402,000 |
On that example run the realistic trigger set cost ~22% of text-pipeline
throughput (4,270 -> 3,320 lines/sec), and the run-to-run spread stayed around
2% - comfortably inside the 10% gate's noise budget. Your own machine will land
somewhere else entirely; that is expected and is exactly why the numbers are not
committed as canonical.
The example was captured on an AMD Ryzen 7 9800X3D / Ubuntu 24.04 / Qt 6.12.0 /
GCC 11.5 **ASan-instrumented, offscreen** functional-test build. ASan and the
offscreen platform dominate the absolute figures (a release build is far faster
and leaner, and `peak_rss_kb` is heavily inflated by ASan shadow memory), which
is another reason to read these only as relative, same-config references.
## Caveats
- Always compare **same machine, same build configuration**. The functional-test
build turns AddressSanitizer on for non-Windows; comparing an ASan build to a
release build, or across hardware, is meaningless.
- The whole corpus is fed as one `loopbackTest()` packet per pass rather than in
network-sized chunks; this measures processing cost, not socket delivery.
- Always compare full-binary runs: `peak_rss_kb` (VmHWM) is process-wide and
monotonic, so filtering to individual test slots changes what it means.
- All benchmark triggers sit at the root of the trigger tree; real profiles nest
most triggers under parent folders, so root iteration is slightly overweighted
relative to real workloads - irrelevant for a relative gate.
- Run order can bias results thermally: whichever binary runs second may execute
on a warmer, throttled CPU, nudging its numbers down. When a comparison lands
close to the threshold, re-run with the order swapped (or let the machine cool)
before trusting a borderline verdict.

252
test/compare-perf-baseline.py Executable file
View file

@ -0,0 +1,252 @@
#!/usr/bin/env python3
"""Compare two PipelineBenchmark runs and gate on throughput regressions.
Absolute benchmark numbers are meaningless across machines, so the only valid
comparison is an OLDER vs a NEWER Mudlet built and run on the SAME machine (the
libmudlet refactor's 10% throughput-loss gate, issue #9011). This reads the
`METRIC <name> <value>` lines PipelineBenchmark prints, computes per-metric
deltas, and exits non-zero if any gated metric regressed past the threshold.
Usage:
# run two already-built binaries:
test/compare-perf-baseline.py --run before-build/.../PipelineBenchmark \\
after-build/.../PipelineBenchmark
# or compare two captured METRIC dumps:
test/compare-perf-baseline.py before.txt after.txt
Exit codes: 0 = within threshold, 1 = a gated metric regressed, 2 = usage error
or the two runs are not comparable. This script is the arbiter of the gate, so it
fails loud (exit 2) rather than silently passing on anything it cannot trust: a
missing or unparseable gated metric, a missing invariant, a non-positive
baseline, or a --gate name that matches no metric.
"""
import argparse
import math
import os
import subprocess
import sys
# Fixed properties of the corpus/trigger set plus the build flavour; if any
# differ, the two runs used different harnesses or build configurations and the
# comparison is invalid - so we abort. build_asan guards against comparing an
# ASan build to a release build, whose absolute numbers are incomparable.
INVARIANTS = ("text_corpus_lines", "text_corpus_bytes", "trigger_count", "build_asan")
# Gated by default: throughput (lines/sec) for the text and trigger pipelines.
# trigger_overhead_ms is intentionally NOT here - it is a difference of two noisy
# best-passes (up to ~16% run-to-run worst case, wider than the 10% gate), so it
# would fire on noise. It stays emitted and reportable, and can be gated
# explicitly with --gate trigger_overhead_ms when a change targets matching.
DEFAULT_GATE = ("text_lines_per_sec", "trigger_lines_per_sec")
# Wall-clock ceiling for a single benchmark run under --run. The ASan/offscreen
# functional-test build feeds a huge corpus several times, so this is generous.
RUN_TIMEOUT_SECONDS = 1200
def fail(message):
"""Abort with exit code 2: a usage error or two runs that cannot be compared."""
sys.stderr.write(f"error: {message}\n")
sys.exit(2)
def classify(name):
"""Return 'higher', 'lower', or 'invariant' for how to read a metric."""
if name in INVARIANTS:
return "invariant"
if name.endswith("_per_sec"):
return "higher" # throughput: bigger is better
if name.endswith("_ms") or name.endswith("_kb"):
return "lower" # time / memory: smaller is better
return "info"
def parse_metrics(text, source):
"""Parse `METRIC <name> <value>` lines, failing hard on anything malformed.
Any line whose first whitespace-token is exactly `METRIC` must parse fully:
exactly three tokens, a finite numeric value, and no duplicate name.
Silently dropping such a line (a NaN/Inf value, a comma decimal, a
concatenated capture) would let a gated metric vanish and the gate pass by
default - the exact failure mode this arbiter must never have.
"""
metrics = {}
for raw in text.splitlines():
line = raw.strip()
if not line.startswith("METRIC"):
continue
parts = line.split()
if parts[0] != "METRIC":
continue # e.g. a "METRICS ..." log line, not one of ours
if len(parts) != 3:
fail(f"{source}: malformed METRIC line {raw!r} (expected 'METRIC <name> <value>')")
name, raw_value = parts[1], parts[2]
try:
value = float(raw_value)
except ValueError:
fail(f"{source}: METRIC {name} has a non-numeric value {raw_value!r}")
if not math.isfinite(value):
fail(f"{source}: METRIC {name} value {raw_value!r} is not a finite number")
if name in metrics:
fail(f"{source}: METRIC {name} appears more than once")
metrics[name] = value
return metrics
def run_binary(path):
if not os.path.isfile(path):
fail(f"{path} is not a file")
if not os.access(path, os.X_OK):
fail(f"{path} is not an executable benchmark binary")
env = dict(os.environ)
env.setdefault("QT_QPA_PLATFORM", "offscreen")
env.setdefault("ASAN_OPTIONS", "detect_leaks=0")
print(f"running {path} ...", file=sys.stderr)
try:
result = subprocess.run(
[os.path.abspath(path)],
capture_output=True,
text=True,
env=env,
timeout=RUN_TIMEOUT_SECONDS,
)
except subprocess.TimeoutExpired:
fail(f"{path} did not finish within {RUN_TIMEOUT_SECONDS}s")
if result.returncode != 0:
sys.stderr.write(result.stdout)
sys.stderr.write(result.stderr)
fail(f"{path} exited with {result.returncode}")
return result.stdout
def load(source, run):
if run:
return parse_metrics(run_binary(source), source)
try:
with open(source, encoding="utf-8") as handle:
return parse_metrics(handle.read(), source)
except OSError as error:
fail(f"cannot read {source}: {error}")
def check_invariants(before, after):
for name in INVARIANTS:
in_before = name in before
in_after = name in after
if not in_before or not in_after:
missing = "before" if not in_before else "after"
fail(
f"invariant {name} is missing from the {missing} run - the two runs are not from "
"the same PipelineBenchmark harness/build and cannot be compared."
)
if before[name] != after[name]:
fail(
f"{name} differs ({before[name]:g} vs {after[name]:g}) - the two runs used "
"different corpora, trigger sets or build configurations and cannot be compared. "
"Rebuild both trees from the same PipelineBenchmark harness, built the same way."
)
def compare(before, after, threshold, gate):
rows = []
failed = False
for name in sorted(set(before) | set(after)):
kind = classify(name)
if kind == "invariant":
continue
gated = name in gate
if name not in before or name not in after:
if gated:
missing = "before" if name not in before else "after"
fail(f"gated metric {name} is missing from the {missing} run - cannot evaluate the gate.")
rows.append((name, "-", "MISSING", ""))
continue
old, new = before[name], after[name]
if old <= 0:
if gated:
fail(
f"gated metric {name} has a non-positive 'before' value ({old:g}); a valid "
"throughput/time baseline must be greater than zero, so the runs are not comparable."
)
rows.append((name, f"{old:g} -> {new:g}", "SKIP", "before <= 0"))
continue
change = (new / old) - 1.0 # signed fractional change, after vs before
if kind == "higher":
regressed = change < -threshold
delta = f"{change * 100:+.1f}%"
elif kind == "lower":
regressed = change > threshold
delta = f"{change * 100:+.1f}% (lower is better)"
else:
regressed = False
delta = f"{change * 100:+.1f}%"
if gated and regressed:
status = "FAIL"
failed = True
elif gated:
status = "PASS"
elif regressed:
status = "warn"
else:
status = "info"
rows.append((name, f"{old:g} -> {new:g}", status, delta))
return rows, failed
def main():
parser = argparse.ArgumentParser(
description="Compare two PipelineBenchmark runs (older vs newer Mudlet, same machine).",
epilog="See docs/libmudlet-perf-baseline.md for the full before/after workflow.",
)
parser.add_argument("before", help="'before' METRIC file, or benchmark binary with --run")
parser.add_argument("after", help="'after' METRIC file, or benchmark binary with --run")
parser.add_argument("--run", action="store_true", help="treat the two arguments as benchmark binaries to run")
parser.add_argument("--threshold", type=float, default=0.10, help="max tolerated fractional regression (default 0.10); a value >= 1 is read as a percentage")
parser.add_argument("--gate", default=",".join(DEFAULT_GATE), help="comma-separated metrics that fail the run")
args = parser.parse_args()
threshold = args.threshold
if threshold <= 0:
fail("--threshold must be greater than 0")
if threshold >= 1:
sys.stderr.write(f"note: --threshold {threshold:g} looks like a percentage; reading it as {threshold / 100:g} ({threshold:g}%).\n")
threshold /= 100.0
gate = {name.strip() for name in args.gate.split(",") if name.strip()}
before = load(args.before, args.run)
after = load(args.after, args.run)
if not before or not after:
fail("no METRIC lines found in one of the runs")
known = set(before) | set(after)
unknown_gates = sorted(name for name in gate if name not in known)
if unknown_gates:
fail(f"--gate names not found in either run: {', '.join(unknown_gates)} - check for a typo.")
check_invariants(before, after)
rows, failed = compare(before, after, threshold, gate)
name_width = max([len("metric")] + [len(row[0]) for row in rows])
value_width = max([len("before -> after")] + [len(row[1]) for row in rows])
print(f"Regression gate: {threshold * 100:.0f}% gated metrics: {', '.join(sorted(gate))}\n")
print(f"{'metric'.ljust(name_width)} {'before -> after'.ljust(value_width)} status delta")
print(f"{'-' * name_width} {'-' * value_width} ------ -----")
for name, value, status, delta in rows:
print(f"{name.ljust(name_width)} {value.ljust(value_width)} {status:<6} {delta}")
print()
if failed:
print(f"FAIL: at least one gated metric lost more than {threshold * 100:.0f}%.")
return 1
print(f"PASS: all gated metrics stayed within {threshold * 100:.0f}%.")
return 0
if __name__ == "__main__":
sys.exit(main())

20
test/functional_tests/CMakeLists.txt
View file

@ -37,6 +37,26 @@ add_dependencies(UndoServerWrapReplay ${LIB_MUDLET_TARGET})
target_link_libraries(UndoServerWrapReplay PRIVATE Qt6::Test ${LIB_MUDLET_TARGET})
set_target_properties(UndoServerWrapReplay PROPERTIES ENABLE_EXPORTS ON)
# Report-only perf harness for manual before/after comparisons
# (test/compare-perf-baseline.py runs the built binary directly). Built with the
# functional tests so a binary always exists, but kept OUT of the default ctest
# suite: it feeds a huge fixed corpus many times and makes no timing assertions,
# so running it every CI pipeline only burns minutes. -DREGISTER_PERF_BENCHMARK=ON
# also registers it with ctest.
option(REGISTER_PERF_BENCHMARK "Register the report-only PipelineBenchmark with ctest (it is built either way)" OFF)
add_executable(PipelineBenchmark PipelineBenchmark.cpp ${FUNCTIONAL_TEST_UTILS})
add_dependencies(PipelineBenchmark ${LIB_MUDLET_TARGET})
target_link_libraries(PipelineBenchmark PRIVATE Qt6::Test ${LIB_MUDLET_TARGET})
set_target_properties(PipelineBenchmark PROPERTIES ENABLE_EXPORTS ON)
if(REGISTER_PERF_BENCHMARK)
add_test(NAME PipelineBenchmark COMMAND $<TARGET_FILE:PipelineBenchmark>)
# large corpus fed repeatedly, so allow a generous timeout
set_tests_properties(PipelineBenchmark PROPERTIES
ENVIRONMENT "QT_QPA_PLATFORM=offscreen;ASAN_OPTIONS=detect_leaks=0"
LABELS "functional"
TIMEOUT 600)
endif()
foreach(test_file ${FUNCTIONAL_TEST_SOURCES})
get_filename_component(test_name ${test_file} NAME_WE)
add_executable(${test_name} ${test_file} ${FUNCTIONAL_TEST_UTILS})

512
test/functional_tests/PipelineBenchmark.cpp Normal file
View file

@ -0,0 +1,512 @@
/***************************************************************************
* Copyright (C) 2026 by Vadim Peretokin - vadim.peretokin@mudlet.org *
* *
* This program is free software; you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation; either version 2 of the License, or *
* (at your option) any later version. *
* *
* This program is distributed in the hope that it will be useful, *
* but WITHOUT ANY WARRANTY; without even the implied warranty of *
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the *
* GNU General Public License for more details. *
* *
* You should have received a copy of the GNU General Public License *
* along with this program; if not, write to the *
* Free Software Foundation, Inc., *
* 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. *
***************************************************************************/
/*
* Report-only performance baseline for the text and trigger pipelines, for the
* libmudlet refactor's "no more than 10% throughput loss" gate (issue #9011).
*
* Absolute numbers are meaningless across machines, so nothing is asserted on
* timing and no baseline is committed: the gate is enforced by comparing an
* older and a newer build of this binary on the SAME machine with
* test/compare-perf-baseline.py. The benchmark feeds a fixed, deterministic
* corpus through the production cTelnet::loopbackTest() path and prints one
* `METRIC <name> <value>` line per measurement.
*
* Built with the functional tests but deliberately NOT registered with ctest by
* default (report-only and slow); run it directly, or configure with
* -DREGISTER_PERF_BENCHMARK=ON to also get it under ctest:
* QT_QPA_PLATFORM=offscreen ./PipelineBenchmark
*
* Companion for the live-GUI display/echo path is the Stressinator display
* package; see docs/libmudlet-perf-baseline.md.
*/
#include <QtTest/QtTest>
#include <algorithm>
#include <clocale>
#include <cstdio>
#include <limits>
#include <random>
// Whether this binary is AddressSanitizer-instrumented. Emitted as an invariant
// so the compare script refuses an ASan-vs-release comparison (their absolute
// numbers are incomparable). Clang reports it through __has_feature; GCC through
// __SANITIZE_ADDRESS__ (and any Qt __has_feature shim harmlessly returns 0, so
// the GCC path still catches it).
#if defined(__has_feature)
#if __has_feature(address_sanitizer)
#define BENCH_BUILD_ASAN 1
#endif
#endif
#if !defined(BENCH_BUILD_ASAN) && defined(__SANITIZE_ADDRESS__)
#define BENCH_BUILD_ASAN 1
#endif
#ifndef BENCH_BUILD_ASAN
#define BENCH_BUILD_ASAN 0
#endif
#include "Host.h"
#include "MudletInstanceCoordinator.h"
#include "TLuaInterpreter.h"
#include "TMainConsole.h"
#include "TTrigger.h"
#include "TelnetServerStub.h"
#include "ctelnet.h"
#include "dlgConnectionProfiles.h"
#include "mudlet.h"
extern void qInitResources_mudlet();
extern void qInitResources_qm();
extern void qInitResources_additional_splash_screens();
extern void qInitResources_mudlet_fonts_common();
extern void qInitResources_mudlet_fonts_posix();
static void initializeQRCResources();
class PipelineBenchmark : public QObject
{
Q_OBJECT
private:
TelnetServerStub* mpServer = nullptr;
const QString mHostname = qsl("Perf-Baseline-Host");
const QString mLocalhost = qsl("localhost");
quint16 mPort = 0;
// Both phases feed these identical bytes, so text and trigger numbers are
// directly comparable.
QByteArray mCorpus;
int mCorpusLines = 0;
qint64 mCorpusBytes = 0;
double mTextBestPassSeconds = 0.0;
// Report the FASTEST pass, not the average: the least-disturbed pass isolates
// intrinsic speed from transient CPU contention (this often runs on a shared/CI
// box), which is what a before/after gate wants. More passes raise the chance
// one lands in a clean window; TConsole's 10 000-line scrollback cap bounds
// memory regardless of corpus size.
static constexpr int kCorpusLines = 25000;
static constexpr int kFeedPasses = 6;
// Seeded with a constant so the corpus bytes are identical on every run and
// every machine; one line per '\n' keeps the processed-line count exact.
static QByteArray generateCorpus(int lines, int& outLineCount)
{
std::mt19937 rng(0xC0FFEEu);
auto pick = [&rng](int n) {
return static_cast<int>(rng() % static_cast<unsigned>(n));
};
// Varied building blocks so substring/regex triggers have realistic text
// to match (and mostly miss) against.
static const char* const rooms[] = {"Village Square", "Dark Forest", "Ancient Tower", "Misty Harbour", "Goblin Warren"};
static const char* const actors[] = {"Gandalf", "Aragorn", "Legolas", "Gimli", "Frodo"};
static const char* const foes[] = {"orc", "goblin", "troll", "wraith", "spider"};
static const char* const items[] = {"a rusty sword", "a wooden shield", "a healing potion", "a silver ring", "a torn map"};
QByteArray out;
out.reserve(static_cast<qsizetype>(lines) * 96);
int count = 0;
for (int i = 0; i < lines; ++i) {
switch (pick(11)) {
case 0:
out += "You are standing in a dark forest. The trees tower above you.";
break;
case 1:
out += "\x1b[1;31mThe ";
out += foes[pick(5)];
out += " hits you for ";
out += QByteArray::number(pick(40) + 1);
out += " damage!\x1b[0m";
break;
case 2:
out += "\x1b[32mThe ";
out += rooms[pick(5)];
out += "\x1b[0m";
break;
case 3:
out += "\x1b[36m";
out += actors[pick(5)];
out += " tells you 'meet me at the tower'\x1b[0m";
break;
case 4:
out += "You gain ";
out += QByteArray::number(pick(500) + 1);
out += " experience points.";
break;
case 5:
out += "The caf\xc3\xa9 serves cr\xc3\xa8me br\xc3\xbbl\xc3\xa9"
"e. \xe6\x97\xa5\xe6\x9c\xac\xe8\xaa\x9e \xe2\x98\xba";
break;
case 6:
out += "\x1b[33mHP: ";
out += QByteArray::number(pick(100) + 1);
out += "/100 MP: ";
out += QByteArray::number(pick(50) + 1);
out += "/50\x1b[0m";
break;
case 7:
out += "You are carrying: ";
out += items[pick(5)];
out += ", ";
out += items[pick(5)];
out += ", and ";
out += QByteArray::number(pick(100));
out += " gold coins.";
break;
case 8:
out += "\x1b[38;5;208mA glowing ember drifts past the ";
out += rooms[pick(5)];
out += ".\x1b[0m";
break;
case 9:
// One long single-line paragraph, to force word-wrap passes the
// short templates never exercise.
out += "The ancient library stretches away in every direction, its towering shelves crammed with "
"mouldering tomes, cracked scrolls and curiosities gathered across a hundred forgotten ages; "
"dust drifts through the amber shafts of light that spill from the high stained-glass windows, "
"and somewhere far above, unseen, the slow tick of a great clock marks out the patient centuries "
"as you catch your breath and let your gaze wander over the winding aisles ahead.";
break;
default:
out += "A gentle breeze carries the scent of pine and distant woodsmoke across the clearing "
"as you catch your breath and survey the winding path ahead.";
break;
}
out += "\r\n";
++count;
}
outLineCount = count;
return out;
}
// A realistic ~three-dozen always-active trigger mix. Some patterns never
// match, so the miss path is costed too. Lua-code matchers are excluded and
// every trigger carries an empty script, so a match runs the full regex +
// capture path (the cost we want) but TTrigger::execute() returns before any
// Lua runs - keeping Lua execution and buffer pollution out of the timed path.
// Prompt triggers are omitted: they need a GA signal a loopback feed cannot send.
int installTriggerSet(Host* host, bool& allOk)
{
int n = 0;
auto addKind = [&](const QStringList& patterns, int kind, bool multiline) {
QList<int> kinds;
kinds.reserve(patterns.size());
for (int i = 0; i < patterns.size(); ++i) {
kinds << kind;
}
auto* pT = new TTrigger(qsl("bench_%1").arg(n), patterns, kinds, multiline, host);
pT->setIsFolder(false);
pT->setTemporary(false);
pT->setConditionLineDelta(5);
pT->setIsActive(true);
allOk = pT->registerTrigger() && allOk;
allOk = pT->setScript(QString()) && allOk;
allOk = pT->state() && allOk;
++n;
};
auto addColor = [&](int ansiFg, int ansiBg) {
auto* pT = new TTrigger(nullptr, host);
pT->setIsFolder(false);
pT->setTemporary(false);
allOk = pT->setupTmpColorTrigger(ansiFg, ansiBg) && allOk;
pT->setIsActive(true);
allOk = pT->registerTrigger() && allOk;
allOk = pT->setScript(QString()) && allOk;
allOk = pT->state() && allOk;
pT->setName(qsl("bench_%1").arg(n));
++n;
};
for (const QString& s :
{qsl("forest"), qsl("orc"), qsl("gold"), qsl("experience"), qsl("sword"), qsl("tower"), qsl("damage"), qsl("coins"), qsl("café"), qsl("Square"), qsl("dragon"), qsl("teleport")}) {
addKind({s}, REGEX_SUBSTRING, false);
}
for (const QString& r : {qsl("^(\\w+) tells you '(.+)'$"),
qsl("You gain (\\d+) experience"),
qsl("hits you for (\\d+) damage"),
qsl("HP: (\\d+)/(\\d+) MP: (\\d+)/(\\d+)"),
qsl("carrying: (.+)$"),
qsl("(\\d+) gold coins"),
qsl("The (\\w+ \\w+)"),
qsl("^A glowing (\\w+)"),
qsl("whisper from (\\w+):"),
qsl("^\\[(\\d{2}):(\\d{2})\\]"),
qsl("reaches level (\\d+)"),
qsl("(\\w+) arrives from the (\\w+)")}) {
addKind({r}, REGEX_PERL, false);
}
for (const QString& s : {qsl("You are"), qsl("The"), qsl("HP:"), qsl("You gain")}) {
addKind({s}, REGEX_BEGIN_OF_LINE_SUBSTRING, false);
}
addColor(1, TTrigger::scmIgnored);
addColor(2, TTrigger::scmIgnored);
addColor(3, TTrigger::scmIgnored);
addColor(6, TTrigger::scmIgnored);
addKind({qsl("The (\\w+) hits you"), qsl("damage")}, REGEX_PERL, true);
addKind({qsl("(\\w+) tells you"), qsl("tower")}, REGEX_PERL, true);
return n;
}
// loopbackTest() writes NUL bytes up to two past the data end, so the corpus
// is over-reserved in initTestCase().
double feedCorpusBestPass(Host* host, int passes)
{
double best = std::numeric_limits<double>::max();
for (int i = 0; i < passes; ++i) {
QElapsedTimer timer;
timer.start();
host->mTelnet.loopbackTest(mCorpus);
best = std::min(best, timer.nsecsElapsed() / 1.0e9);
}
return best;
}
static void emitMetric(const char* name, double value)
{
std::printf("METRIC %s %.2f\n", name, value);
std::fflush(stdout);
}
static void emitMetric(const char* name, qint64 value)
{
std::printf("METRIC %s %lld\n", name, value);
std::fflush(stdout);
}
// Process-wide peak RSS in kB (VmHWM never decreases). /proc pseudo-files
// report a size of 0, so QFile::atEnd() is immediately true and readLine()
// loops never start - read it all in one go.
static qint64 readPeakRssKb()
{
#if defined(Q_OS_LINUX)
QFile status(qsl("/proc/self/status"));
if (!status.open(QIODevice::ReadOnly | QIODevice::Text)) {
return -1;
}
const QList<QByteArray> lines = status.readAll().split('\n');
for (const QByteArray& line : lines) {
if (line.startsWith("VmHWM:")) {
const QList<QByteArray> parts = line.simplified().split(' ');
if (parts.size() >= 2) {
return parts.at(1).toLongLong();
}
}
}
return -1;
#else
return -1;
#endif
}
private slots:
void initTestCase()
{
// QApplication's construction adopts the environment locale, which on some
// machines makes printf("%f") emit comma decimals the compare script cannot
// parse. Force C numeric formatting for every METRIC line, independent of
// whatever the environment or Lua startup leaves LC_NUMERIC at.
std::setlocale(LC_NUMERIC, "C");
initializeQRCResources();
mCorpus = generateCorpus(kCorpusLines, mCorpusLines);
mCorpusBytes = mCorpus.size();
// loopbackTest() writes NUL bytes past the data end; reserve slack so that
// stays within the allocation.
mCorpus.reserve(mCorpus.size() + 16);
// An invariant, emitted here so it is present regardless of which bench
// slots run: the compare script rejects an ASan-vs-release comparison.
emitMetric("build_asan", static_cast<qint64>(BENCH_BUILD_ASAN));
qInfo().nospace() << "Corpus: " << mCorpusLines << " lines, " << mCorpusBytes << " bytes";
}
void init()
{
mpServer = new TelnetServerStub(qApp);
// Ephemeral port (0) so parallel worktree runs never collide; read the
// actual port back afterwards.
mpServer->start(mLocalhost, 0);
mPort = mpServer->serverPort();
mudlet::start();
mudlet::self()->setupConfig();
mudlet::self()->takeOwnershipOfInstanceCoordinator(std::make_unique<MudletInstanceCoordinator>("MudletInstanceCoordinator"));
mudlet::self()->init();
mudlet::self()->setStorePasswordsSecurely(false);
deleteProfileDirectory(mHostname);
}
void cleanup()
{
delete mpServer;
mpServer = nullptr;
deleteProfileDirectory(mHostname);
delete mudlet::self();
}
void benchTextPipeline()
{
Host* host = startProfile();
QVERIFY(host);
const double seconds = feedCorpusBestPass(host, kFeedPasses);
mTextBestPassSeconds = seconds;
// A silently-disconnected pipeline would report absurdly good numbers, so
// prove data flowed: the console must sit near its 10 000-line scrollback cap.
const int bufferedLines = host->mpConsole->buffer.getLastLineNumber();
QVERIFY2(bufferedLines > 1000, qPrintable(qsl("console buffer only holds %1 lines - the pipeline did not process the corpus").arg(bufferedLines)));
emitMetric("text_corpus_lines", static_cast<qint64>(mCorpusLines));
emitMetric("text_corpus_bytes", mCorpusBytes);
emitMetric("text_lines_per_sec", mCorpusLines / seconds);
emitMetric("text_mb_per_sec", (mCorpusBytes / 1.0e6) / seconds);
emitMetric("text_best_pass_ms", seconds * 1000.0);
}
void benchTriggerEngine()
{
Host* host = startProfile();
QVERIFY(host);
bool triggersOk = true;
const int triggerCount = installTriggerSet(host, triggersOk);
QVERIFY2(triggerCount > 0, "no triggers were installed");
QVERIFY2(triggersOk, "a trigger failed to compile, register or take its script");
const double seconds = feedCorpusBestPass(host, kFeedPasses);
const int bufferedLines = host->mpConsole->buffer.getLastLineNumber();
QVERIFY2(bufferedLines > 1000, qPrintable(qsl("console buffer only holds %1 lines - the pipeline did not process the corpus").arg(bufferedLines)));
// Untimed sentinel proving TriggerUnit consumes what the loopback path
// feeds - a disconnected trigger engine would just flatter the timed numbers.
auto* sentinel = new TTrigger(qsl("bench_sentinel"), {qsl("__bench_sentinel__")}, {REGEX_SUBSTRING}, false, host);
sentinel->setIsFolder(false);
sentinel->setTemporary(false);
sentinel->setIsActive(true);
QVERIFY(sentinel->registerTrigger());
QVERIFY(sentinel->setScript(qsl("benchSentinelFired = true")));
QVERIFY(sentinel->state());
QByteArray probe{"__bench_sentinel__\r\n"};
probe.reserve(probe.size() + 16);
host->mTelnet.loopbackTest(probe);
QVERIFY2(host->getLuaInterpreter()->compileAndExecuteScript(qsl("assert(benchSentinelFired)")), "sentinel trigger did not fire - the trigger engine is not seeing pipeline data");
emitMetric("trigger_count", static_cast<qint64>(triggerCount));
emitMetric("trigger_lines_per_sec", mCorpusLines / seconds);
emitMetric("trigger_mb_per_sec", (mCorpusBytes / 1.0e6) / seconds);
emitMetric("trigger_best_pass_ms", seconds * 1000.0);
if (mTextBestPassSeconds > 0.0) {
// Trigger throughput includes the text-pipeline cost, which dilutes a
// matcher-only regression ~4x; subtracting isolates it (valid because
// both phases feed identical bytes).
emitMetric("trigger_overhead_ms", (seconds - mTextBestPassSeconds) * 1000.0);
}
}
// VmHWM is process-wide and monotonic, so reading it after the feed phases
// captures the true peak for the whole run.
void benchPeakMemory()
{
Host* host = startProfile();
QVERIFY(host);
// Feed one pass so the peak still reflects pipeline work when this slot
// runs on its own.
feedCorpusBestPass(host, 1);
// Skip the metric entirely when the read fails (non-Linux, or /proc
// unavailable) rather than emitting a bogus -1 the compare script would
// read as a real value.
const qint64 peakRssKb = readPeakRssKb();
if (peakRssKb >= 0) {
emitMetric("peak_rss_kb", peakRssKb);
}
}
private:
// Mirrors the profile-creation helper the other functional tests use.
Host* startProfile()
{
const QString port = QString::number(mPort);
QTimer::singleShot(0, qApp, [this, port]() {
mudlet::self()->startAutoLogin({});
QTest::qWait(100);
QTest::mouseClick(mudlet::self()->mpConnectionDialog->new_profile_button, Qt::LeftButton);
QTest::qWait(100);
QTest::keyClicks(QApplication::focusWidget(), mHostname);
QTest::qWait(100);
QTest::keyClick(QApplication::focusWidget(), Qt::Key_Tab);
QTest::qWait(100);
QTest::keyClicks(QApplication::focusWidget(), mLocalhost);
QTest::qWait(100);
QTest::keyClick(QApplication::focusWidget(), Qt::Key_Tab);
QTest::qWait(100);
QTest::keyClicks(QApplication::focusWidget(), port);
QTest::qWait(100);
QTest::keyClick(QApplication::focusWidget(), Qt::Key_Return);
});
QSignalSpy loaded(mudlet::self(), &mudlet::signal_profileLoaded);
if (!loaded.wait(5000)) {
qWarning("Profile took too long to load");
return nullptr;
}
Host* host = mudlet::self()->getActiveHost();
if (!host) {
qWarning("No active host");
return nullptr;
}
QSignalSpy connected(&(host->mTelnet), &cTelnet::signal_connected);
if (!connected.wait(3000)) {
qWarning("Could not connect to the stub");
return nullptr;
}
return host;
}
void deleteProfileDirectory(const QString& profileName)
{
const QString path = mudlet::getMudletPath(enums::profileHomePath, profileName);
QDir dir(path);
if (dir.exists()) {
dir.removeRecursively();
}
}
};
static void initializeQRCResources()
{
#ifdef INCLUDE_VARIABLE_SPLASH_SCREEN
qInitResources_additional_splash_screens();
#endif
#ifdef INCLUDE_FONTS
qInitResources_mudlet_fonts_common();
#if defined(Q_OS_LINUX) || defined(Q_OS_FREEBSD)
qInitResources_mudlet_fonts_posix();
#endif
#endif
qInitResources_mudlet();
qInitResources_qm();
}
#include "PipelineBenchmark.moc"
QTEST_MAIN(PipelineBenchmark)