mirror of
https://github.com/microsoft/GSL
synced 2026-08-26 04:23:04 -04:00
Refactor benchmark CI configuration and add span benchmark comment workflow
- Updated span_benchmark.yml to streamline job configurations and improve artifact handling. - Introduced span_benchmark_comment.yml to automate PR comment updates with benchmark results. - Enhanced CMakeLists.txt to support benchmark builds with new GSL_BENCHMARK option. - Removed README.md from benchmark directory and adjusted check_regression.py for cleaner report generation. - Modified span_bench.cpp to include necessary headers for benchmarking.
This commit is contained in:
Mohammad Abdul Gafoor
parent
ca7905d901
commit
ff48618829
7 changed files with 154 additions and 517 deletions
333
.github/workflows/span_benchmark.yml
vendored
333
.github/workflows/span_benchmark.yml
vendored
|
|
@ -1,256 +1,94 @@
|
|||
# .github/workflows/span_benchmark.yml
|
||||
#
|
||||
# Benchmarks gsl::span vs std::span on every PR across all supported compilers.
|
||||
# Strategy: in-run ratio (gsl_ns / std_ns) — noise-resistant because both
|
||||
# spans run in the same process, same machine, same moment.
|
||||
#
|
||||
# All dependencies (google-benchmark v1.9.0, GSL v4.1.0) are pulled via
|
||||
# FetchContent inside the repo's CMakeLists.txt — no separate install steps.
|
||||
# The build/_deps directory is cached, keyed on CMakeLists.txt hash, so
|
||||
# repeated runs don't re-clone anything.
|
||||
#
|
||||
# Three separate job groups (one per OS) because `runs-on` can't share a
|
||||
# matrix with OS-specific shell/path differences cleanly:
|
||||
#
|
||||
# benchmark-linux → ubuntu-latest → GCC-13, GCC-14, Clang-17, Clang-18
|
||||
# benchmark-windows → windows-latest → MSVC 2022, clang-cl
|
||||
# benchmark-macos → macos-14 → Apple Clang (Xcode latest)
|
||||
#
|
||||
# A final `comment` job waits for all three groups, collects every JSON
|
||||
# artifact, runs check_regression.py, and posts (or updates) one PR comment.
|
||||
|
||||
|
||||
name: Span Benchmark
|
||||
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
|
||||
# Cancel stale runs when a new commit is pushed to the same PR.
|
||||
concurrency:
|
||||
group: span-bench-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
|
||||
|
||||
jobs:
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# LINUX — GCC 13/14 · Clang 17/18 (× C++20 and C++23)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
benchmark-linux:
|
||||
name: Linux / ${{ matrix.label }}
|
||||
name: Linux / ${{ format('{0}-cpp{1}', matrix.cxx, matrix.cppstd) }}
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
# ── GCC ────────────────────────────────────────────────────────
|
||||
- label: GCC-13-cpp20
|
||||
cxx: g++-13
|
||||
cppstd: 20
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
- label: GCC-14-cpp20
|
||||
cxx: g++-14
|
||||
cppstd: 20
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
- label: GCC-14-cpp23
|
||||
cxx: g++-14
|
||||
cppstd: 23
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
# ── Clang ──────────────────────────────────────────────────────
|
||||
- label: Clang-17-cpp20
|
||||
cxx: clang++-17
|
||||
cppstd: 20
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
- label: Clang-18-cpp20
|
||||
cxx: clang++-18
|
||||
cppstd: 20
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
- label: Clang-18-cpp23
|
||||
cxx: clang++-18
|
||||
cppstd: 23
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
# ── Clang + -fbounds-safety (core motivation of issue #1167) ───
|
||||
# Safe Buffers enforcement is what triggered this whole tracking effort.
|
||||
# - label: Clang-18-cpp20-bounds-safety
|
||||
# cxx: clang++-18
|
||||
# cppstd: 20
|
||||
# extra_flags: "-fbounds-safety"
|
||||
# ----------------------------------------------------------------------------
|
||||
# ⚠️ Note: As of early 2026, the feature is not yet fully available or stable in mainline, official releases (like Clang 20, 21, or 22).
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
|
||||
cxx: [g++-13, g++-14, clang++-16, clang++-17, clang++-18]
|
||||
cppstd: [20, 23]
|
||||
exclude:
|
||||
- cxx: g++-13
|
||||
cppstd: 23
|
||||
- cxx: clang++-17
|
||||
cppstd: 23
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
|
||||
# ── Compiler install ───────────────────────────────────────────────
|
||||
# GCC-13 and Clang pre-17 are already on ubuntu-latest (24.04).
|
||||
- name: Install GCC 14
|
||||
if: startsWith(matrix.cxx, 'g++-14')
|
||||
run: |
|
||||
sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y g++-14
|
||||
|
||||
|
||||
- name: Install Clang 17
|
||||
if: matrix.cxx == 'clang++-17'
|
||||
run: |
|
||||
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key \
|
||||
| sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
|
||||
echo "deb http://apt.llvm.org/noble/ llvm-toolchain-noble-17 main" \
|
||||
| sudo tee /etc/apt/sources.list.d/llvm-17.list
|
||||
sudo apt-get update -qq && sudo apt-get install -y clang-17
|
||||
|
||||
|
||||
- name: Install Clang 18
|
||||
if: matrix.cxx == 'clang++-18'
|
||||
run: |
|
||||
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key \
|
||||
| sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
|
||||
echo "deb http://apt.llvm.org/noble/ llvm-toolchain-noble-18 main" \
|
||||
| sudo tee /etc/apt/sources.list.d/llvm-18.list
|
||||
sudo apt-get update -qq && sudo apt-get install -y clang-18
|
||||
|
||||
|
||||
# ── FetchContent cache ─────────────────────────────────────────────
|
||||
# Caches the cloned sources for google-benchmark, googletest, and GSL.
|
||||
# Key is the hash of CMakeLists.txt — busts automatically on any
|
||||
# GIT_TAG bump without manual intervention.
|
||||
- name: Cache FetchContent dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: build/_deps
|
||||
key: fetchcontent-linux-${{ matrix.cxx }}-${{ hashFiles('CMakeLists.txt') }}
|
||||
key: fetchcontent-linux-${{ matrix.cxx }}-${{ hashFiles('benchmark/CMakeLists.txt') }}
|
||||
restore-keys: |
|
||||
fetchcontent-linux-${{ matrix.cxx }}-
|
||||
|
||||
|
||||
# ── Configure → Build → Run ────────────────────────────────────────
|
||||
# FetchContent handles benchmark + GSL — no separate install needed.
|
||||
# -DCMAKE_CXX_STANDARD overrides the default C++20 set in CMakeLists
|
||||
# so we can test C++23 configs from the matrix.
|
||||
- name: Configure
|
||||
run: |
|
||||
cmake -S benchmark -B build \
|
||||
cmake -S . -B build \
|
||||
-DGSL_BENCHMARK=ON \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_CXX_COMPILER=${{ matrix.cxx }} \
|
||||
-DCMAKE_CXX_STANDARD=${{ matrix.cppstd }} \
|
||||
-DCMAKE_CXX_FLAGS="${{ matrix.extra_flags }}"
|
||||
|
||||
-DCMAKE_CXX_STANDARD=${{ matrix.cppstd }}"
|
||||
|
||||
- name: Build
|
||||
run: cmake --build build --target span_bench -j$(nproc)
|
||||
|
||||
|
||||
# 10 repetitions → mean + stddev aggregates in the JSON output.
|
||||
# benchmark_report_aggregates_only keeps the file compact.
|
||||
- name: Run benchmark
|
||||
run: |
|
||||
./build/span_bench \
|
||||
--benchmark_format=json \
|
||||
--benchmark_repetitions=10 \
|
||||
--benchmark_report_aggregates_only=true \
|
||||
--benchmark_out=results_${{ matrix.label }}.json
|
||||
|
||||
--benchmark_out=results_${{ format('{0}-cpp{1}', matrix.cxx, matrix.cppstd) }}.json
|
||||
|
||||
- name: Upload results
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: bench-${{ matrix.label }}
|
||||
path: results_${{ matrix.label }}.json
|
||||
name: bench-${{ format('{0}-cpp{1}', matrix.cxx, matrix.cppstd) }}
|
||||
path: results_${{ format('{0}-cpp{1}', matrix.cxx, matrix.cppstd) }}.json
|
||||
retention-days: 7
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# WINDOWS — MSVC 2022 · clang-cl (× C++20 and C++23)
|
||||
#
|
||||
# Notes:
|
||||
# • Uses the Visual Studio 17 2022 generator for both MSVC and clang-cl.
|
||||
# The -T ClangCL toolset switch selects the clang-cl frontend that ships
|
||||
# bundled with VS 2022 — no extra install needed.
|
||||
# • Multi-config generator: output binary is at
|
||||
# build\Release\span_bench.exe (FetchContent flat layout, no subdir).
|
||||
# • FetchContent cache is keyed per-toolset because MSVC and clang-cl
|
||||
# produce ABI-incompatible object files.
|
||||
# • PowerShell is used throughout (shell: pwsh) for consistent quoting.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
benchmark-windows:
|
||||
name: Windows / ${{ matrix.label }}
|
||||
name: Windows / ${{ format('{0}-cpp{1}', matrix.toolset || 'MSVC', matrix.cppstd) }}
|
||||
runs-on: windows-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
# ── MSVC 2022 ──────────────────────────────────────────────────
|
||||
- label: MSVC-2022-cpp20
|
||||
toolset: ""
|
||||
cppstd: 20
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
- label: MSVC-2022-cpp23
|
||||
toolset: ""
|
||||
cppstd: 23
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
# ── clang-cl (bundled with VS 2022) ────────────────────────────
|
||||
- label: clang-cl-cpp20
|
||||
toolset: "ClangCL"
|
||||
cppstd: 20
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
- label: clang-cl-cpp23
|
||||
toolset: "ClangCL"
|
||||
cppstd: 23
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
generator: [ 'Visual Studio 17 2022' ]
|
||||
toolset: [ '', 'ClangCL' ]
|
||||
cppstd: [ 20, 23 ]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
|
||||
# ── FetchContent cache ─────────────────────────────────────────────
|
||||
- name: Cache FetchContent dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: build/_deps
|
||||
key: fetchcontent-windows-${{ matrix.toolset }}-${{ hashFiles('CMakeLists.txt') }}
|
||||
key: fetchcontent-windows-${{ matrix.toolset }}-${{ hashFiles('benchmark/CMakeLists.txt') }}
|
||||
restore-keys: |
|
||||
fetchcontent-windows-${{ matrix.toolset }}-
|
||||
|
||||
|
||||
# ── Configure → Build → Run ────────────────────────────────────────
|
||||
- name: Configure
|
||||
shell: pwsh
|
||||
run: |
|
||||
$tsArg = if ("${{ matrix.toolset }}" -ne "") { @("-T", "${{ matrix.toolset }}") } else { @() }
|
||||
|
||||
|
||||
cmake -S benchmark -B build `
|
||||
-G "Visual Studio 17 2022" @tsArg `
|
||||
-DCMAKE_CXX_STANDARD=${{ matrix.cppstd }} `
|
||||
-DCMAKE_CXX_FLAGS="${{ matrix.extra_flags }}"
|
||||
|
||||
cmake -S . -B build `
|
||||
-DGSL_BENCHMARK=ON `
|
||||
-G "${{ matrix.generator }}" @tsArg `
|
||||
-DCMAKE_CXX_STANDARD=${{ matrix.cppstd }}"
|
||||
|
||||
- name: Build
|
||||
shell: pwsh
|
||||
|
|
@ -258,8 +96,6 @@ jobs:
|
|||
cmake --build build --target span_bench `
|
||||
--config Release -j $env:NUMBER_OF_PROCESSORS
|
||||
|
||||
|
||||
# Multi-config generator places the binary under build\Release\
|
||||
- name: Run benchmark
|
||||
shell: pwsh
|
||||
run: |
|
||||
|
|
@ -267,147 +103,54 @@ jobs:
|
|||
--benchmark_format=json `
|
||||
--benchmark_repetitions=10 `
|
||||
--benchmark_report_aggregates_only=true `
|
||||
--benchmark_out=results_${{ matrix.label }}.json
|
||||
|
||||
--benchmark_out=results_${{ format('{0}-cpp{1}', matrix.toolset || 'MSVC', matrix.cppstd) }}.json
|
||||
|
||||
- name: Upload results
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: bench-${{ matrix.label }}
|
||||
path: results_${{ matrix.label }}.json
|
||||
name: bench-${{ format('{0}-cpp{1}', matrix.toolset || 'MSVC', matrix.cppstd) }}
|
||||
path: results_${{ format('{0}-cpp{1}', matrix.toolset || 'MSVC', matrix.cppstd) }}.json
|
||||
retention-days: 7
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# MACOS — Apple Clang via Xcode (latest) (× C++20 and C++23)
|
||||
#
|
||||
# Notes:
|
||||
# • macos-14 = Apple Silicon (M1). Switch to macos-13 for Intel if needed.
|
||||
# • No extra compiler install — Apple Clang from Xcode is used directly.
|
||||
# • -j uses sysctl (macOS equivalent of nproc).
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
benchmark-macos:
|
||||
name: macOS / ${{ matrix.label }}
|
||||
runs-on: macos-14
|
||||
name: macOS / ${{ format('AppleClang-cpp{0}', matrix.cppstd) }}
|
||||
runs-on: macos-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- label: AppleClang-cpp20
|
||||
cppstd: 20
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
- label: AppleClang-cpp23
|
||||
cppstd: 23
|
||||
extra_flags: ""
|
||||
|
||||
|
||||
cppstd: [ 20, 23 ]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
|
||||
# ── FetchContent cache ─────────────────────────────────────────────
|
||||
- name: Cache FetchContent dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: build/_deps
|
||||
key: fetchcontent-macos-${{ matrix.cppstd }}-${{ hashFiles('CMakeLists.txt') }}
|
||||
key: fetchcontent-macos-${{ matrix.cppstd }}-${{ hashFiles('benchmark/CMakeLists.txt') }}
|
||||
restore-keys: |
|
||||
fetchcontent-macos-${{ matrix.cppstd }}-
|
||||
|
||||
|
||||
# ── Configure → Build → Run ────────────────────────────────────────
|
||||
- name: Configure
|
||||
run: |
|
||||
cmake -S benchmark -B build \
|
||||
cmake -S . -B build \
|
||||
-DGSL_BENCHMARK=ON \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_CXX_STANDARD=${{ matrix.cppstd }} \
|
||||
-DCMAKE_CXX_FLAGS="${{ matrix.extra_flags }}"
|
||||
|
||||
-DCMAKE_CXX_STANDARD=${{ matrix.cppstd }}"
|
||||
|
||||
- name: Build
|
||||
run: cmake --build build --target span_bench -j$(sysctl -n hw.logicalcpu)
|
||||
|
||||
|
||||
- name: Run benchmark
|
||||
run: |
|
||||
./build/span_bench \
|
||||
--benchmark_format=json \
|
||||
--benchmark_repetitions=10 \
|
||||
--benchmark_report_aggregates_only=true \
|
||||
--benchmark_out=results_${{ matrix.label }}.json
|
||||
|
||||
--benchmark_out=results_${{ format('AppleClang-cpp{0}', matrix.cppstd) }}.json
|
||||
|
||||
- name: Upload results
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: bench-${{ matrix.label }}
|
||||
path: results_${{ matrix.label }}.json
|
||||
retention-days: 7
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# COMMENT — collect every result artifact → one PR comment
|
||||
#
|
||||
# Waits for all three OS groups. `if: always()` ensures this runs even when
|
||||
# some benchmark jobs fail — partial results are always reported.
|
||||
# The CI check fails at the very end (after posting) if any regression found.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
comment:
|
||||
name: Post PR comment
|
||||
needs: [benchmark-linux, benchmark-windows, benchmark-macos]
|
||||
runs-on: ubuntu-latest
|
||||
if: always() && github.event_name == 'pull_request'
|
||||
permissions:
|
||||
pull-requests: write
|
||||
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
|
||||
- name: Download all result artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: bench-*
|
||||
merge-multiple: true # flatten all artifacts into CWD
|
||||
|
||||
|
||||
# continue-on-error so the comment step always runs even on regression.
|
||||
- name: Generate regression report
|
||||
id: report
|
||||
run: |
|
||||
python3 benchmark/check_regression.py \
|
||||
--threshold 0.15 \
|
||||
--output comment.md \
|
||||
--workflow-url "https://github.com/${{ github.repository }}/blob/tree/main/.github/workflows/span_benchmark.yml" \
|
||||
results_*.json
|
||||
continue-on-error: true
|
||||
|
||||
|
||||
- name: Find existing bot comment
|
||||
uses: peter-evans/find-comment@v3
|
||||
id: find_comment
|
||||
with:
|
||||
issue-number: ${{ github.event.pull_request.number }}
|
||||
comment-author: github-actions[bot]
|
||||
body-includes: "<!-- span-bench-report -->"
|
||||
|
||||
|
||||
# One comment per PR, updated on every push — not a flood of new ones.
|
||||
- name: Create or update PR comment
|
||||
uses: peter-evans/create-or-update-comment@v4
|
||||
with:
|
||||
comment-id: ${{ steps.find_comment.outputs.comment-id }}
|
||||
issue-number: ${{ github.event.pull_request.number }}
|
||||
body-path: comment.md
|
||||
edit-mode: replace
|
||||
|
||||
|
||||
# Surface the failure visibly in the CI check panel.
|
||||
- name: Fail if regression detected
|
||||
if: steps.report.outcome == 'failure'
|
||||
run: |
|
||||
echo "::error::Performance regression detected. See the PR comment for the full table."
|
||||
exit 1
|
||||
name: bench-${{ format('AppleClang-cpp{0}', matrix.cppstd) }}
|
||||
path: results_${{ format('AppleClang-cpp{0}', matrix.cppstd) }}.json
|
||||
retention-days: 7
|
||||
76
.github/workflows/span_benchmark_comment.yml
vendored
Normal file
76
.github/workflows/span_benchmark_comment.yml
vendored
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
name: Span Benchmark Comment
|
||||
|
||||
on:
|
||||
workflow_run:
|
||||
workflows: ["Span Benchmark"]
|
||||
types: [completed]
|
||||
|
||||
permissions:
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
comment:
|
||||
name: Post PR comment
|
||||
runs-on: ubuntu-latest
|
||||
if: always()
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
# Download all bench-* artifacts from the triggering workflow run.
|
||||
- name: Download all result artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: bench-*
|
||||
merge-multiple: true
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
run-id: ${{ github.event.workflow_run.id }}
|
||||
|
||||
- name: Generate regression report
|
||||
id: report
|
||||
run: |
|
||||
python3 benchmark/check_regression.py \
|
||||
--threshold 0.15 \
|
||||
--output comment.md \
|
||||
results_*.json
|
||||
continue-on-error: true
|
||||
|
||||
# Resolve the PR number from the triggering workflow run.
|
||||
- name: Get PR number
|
||||
id: pr
|
||||
uses: actions/github-script@v7
|
||||
with:
|
||||
script: |
|
||||
const runs = await github.rest.pulls.list({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
state: 'open',
|
||||
});
|
||||
const pr = runs.data.find(p =>
|
||||
p.head.sha === context.payload.workflow_run.head_sha
|
||||
);
|
||||
return pr ? pr.number : null;
|
||||
|
||||
- name: Find existing bot comment
|
||||
if: steps.pr.outputs.result != 'null'
|
||||
uses: peter-evans/find-comment@v3
|
||||
id: find_comment
|
||||
with:
|
||||
issue-number: ${{ steps.pr.outputs.result }}
|
||||
comment-author: github-actions[bot]
|
||||
body-includes: "<!-- span-bench-report -->"
|
||||
|
||||
- name: Create or update PR comment
|
||||
if: steps.pr.outputs.result != 'null'
|
||||
uses: peter-evans/create-or-update-comment@v4
|
||||
with:
|
||||
comment-id: ${{ steps.find_comment.outputs.comment-id }}
|
||||
issue-number: ${{ steps.pr.outputs.result }}
|
||||
body-path: comment.md
|
||||
edit-mode: replace
|
||||
|
||||
- name: Fail if regression detected
|
||||
if: steps.report.outcome == 'failure'
|
||||
run: |
|
||||
echo "::error::Performance regression detected. See the PR comment for the full table."
|
||||
exit 1
|
||||
|
|
@ -10,6 +10,7 @@ string(COMPARE EQUAL ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_SOURCE_DIR} PROJECT_IS_
|
|||
|
||||
option(GSL_INSTALL "Generate and install GSL target" ${PROJECT_IS_TOP_LEVEL})
|
||||
option(GSL_TEST "Build and perform GSL tests" ${PROJECT_IS_TOP_LEVEL})
|
||||
option(GSL_BENCHMARK "Build span benchmarks" ${PROJECT_IS_TOP_LEVEL})
|
||||
|
||||
# The implementation generally assumes a platform that implements C++14 support
|
||||
target_compile_features(GSL INTERFACE "cxx_std_14")
|
||||
|
|
@ -19,6 +20,10 @@ add_subdirectory(include)
|
|||
|
||||
target_sources(GSL INTERFACE $<BUILD_INTERFACE:${GSL_SOURCE_DIR}/GSL.natvis>)
|
||||
|
||||
if(GSL_BENCHMARK)
|
||||
add_subdirectory(benchmark)
|
||||
endif()
|
||||
|
||||
if (GSL_TEST)
|
||||
enable_testing()
|
||||
add_subdirectory(tests)
|
||||
|
|
|
|||
|
|
@ -1,87 +1,57 @@
|
|||
# benchmark/CMakeLists.txt
|
||||
#
|
||||
# Self-contained build for the gsl::span vs std::span benchmark.
|
||||
# All dependencies are fetched automatically via FetchContent:
|
||||
# - google/benchmark v1.9.0
|
||||
# - google/googletest release-1.12.1 (benchmark's internal dep)
|
||||
# - microsoft/GSL v4.1.0
|
||||
#
|
||||
# Usage (from this directory):
|
||||
# cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
|
||||
# cmake --build build --target span_bench
|
||||
# ./build/span_bench --benchmark_format=json --benchmark_repetitions=10 \
|
||||
# --benchmark_report_aggregates_only=true \
|
||||
# --benchmark_out=results.json
|
||||
#
|
||||
# To override the compiler or standard (as CI does):
|
||||
# cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
|
||||
# -DCMAKE_CXX_COMPILER=clang++-18 \
|
||||
# -DCMAKE_CXX_STANDARD=23
|
||||
cmake_minimum_required(VERSION 3.14...3.16)
|
||||
|
||||
cmake_minimum_required(VERSION 3.20)
|
||||
project(GSLBenchmarks LANGUAGES CXX)
|
||||
|
||||
project(gsl_span_benchmark CXX)
|
||||
# ── C++ standard ───────────────────────────────────────────────────────────────
|
||||
# Minimum is 20 because std::span (required for this benchmark) is C++20 only.
|
||||
# If a consumer passes GSL_CXX_STANDARD < 20 we clamp and warn.
|
||||
set(GSL_CXX_STANDARD "20" CACHE STRING "Use c++ standard")
|
||||
|
||||
# ── C++ standard ──────────────────────────────────────────────────────────────
|
||||
# Default C++20 (minimum for std::span). CI overrides via -DCMAKE_CXX_STANDARD.
|
||||
set(CMAKE_CXX_STANDARD 20 CACHE STRING "C++ standard")
|
||||
if(GSL_CXX_STANDARD LESS 20)
|
||||
message(WARNING
|
||||
"GSL_CXX_STANDARD=${GSL_CXX_STANDARD} is too old for the span benchmark "
|
||||
"(std::span requires C++20). Overriding to 20.")
|
||||
set(GSL_CXX_STANDARD "20" CACHE STRING "Use c++ standard" FORCE)
|
||||
endif()
|
||||
|
||||
set(CMAKE_CXX_STANDARD ${GSL_CXX_STANDARD})
|
||||
set(CMAKE_CXX_EXTENSIONS OFF)
|
||||
set(CMAKE_CXX_STANDARD_REQUIRED ON)
|
||||
set(CMAKE_CXX_EXTENSIONS NO)
|
||||
|
||||
set(CMAKE_BUILD_TYPE Release CACHE STRING "Build type")
|
||||
# Makes Visual Studio organise targets into folders
|
||||
set_property(GLOBAL PROPERTY USE_FOLDERS ON)
|
||||
|
||||
# ── FetchContent dependencies ──────────────────────────────────────────────────
|
||||
include(FetchContent)
|
||||
|
||||
# Suppress benchmark's own test suite — we don't need it.
|
||||
# Suppress benchmark's own test suite
|
||||
set(BENCHMARK_ENABLE_TESTING OFF CACHE BOOL "" FORCE)
|
||||
|
||||
FetchContent_Declare(
|
||||
googletest
|
||||
GIT_REPOSITORY https://github.com/google/googletest.git
|
||||
GIT_TAG release-1.12.1
|
||||
GIT_SHALLOW ON
|
||||
)
|
||||
set(BENCHMARK_ENABLE_INSTALL OFF CACHE BOOL "" FORCE)
|
||||
|
||||
FetchContent_Declare(
|
||||
googlebenchmark
|
||||
GIT_REPOSITORY https://github.com/google/benchmark.git
|
||||
GIT_TAG v1.9.0
|
||||
GIT_TAG v1.9.5
|
||||
GIT_SHALLOW ON
|
||||
)
|
||||
FetchContent_MakeAvailable(googlebenchmark)
|
||||
|
||||
FetchContent_MakeAvailable(googletest googlebenchmark)
|
||||
|
||||
# Use the GSL from the local checkout — NOT a pinned release tag.
|
||||
# The benchmark/ folder sits one level below the repo root, so
|
||||
# CMAKE_CURRENT_SOURCE_DIR/.. resolves to the repo root which contains
|
||||
# the real include/gsl/span header being modified by the PR.
|
||||
# This is what makes the CI actually test the PR's changes.
|
||||
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/.. gsl_root)
|
||||
|
||||
# ── Benchmark executable ───────────────────────────────────────────────────────
|
||||
add_executable(span_bench span_bench.cpp)
|
||||
|
||||
# Link benchmark::benchmark only — NOT benchmark::benchmark_main.
|
||||
# span_bench.cpp uses the BENCHMARK_MAIN() macro which expands to its own
|
||||
# main(), so linking benchmark_main would cause a duplicate-symbol linker error.
|
||||
target_link_libraries(span_bench PRIVATE
|
||||
benchmark::benchmark # provides the benchmark framework + runner
|
||||
Microsoft.GSL::GSL # provides gsl::span and friends
|
||||
benchmark::benchmark
|
||||
Microsoft.GSL::GSL
|
||||
)
|
||||
|
||||
set_target_properties(span_bench PROPERTIES FOLDER "benchmarks")
|
||||
|
||||
# ── Optimisation flags ─────────────────────────────────────────────────────────
|
||||
# -O3 and -march=native let the compiler apply the same optimisations to both
|
||||
# gsl::span and std::span, keeping the comparison fair.
|
||||
# -fno-omit-frame-pointer keeps perf/profiling usable if you need it locally.
|
||||
target_compile_options(span_bench PRIVATE
|
||||
$<$<CXX_COMPILER_ID:GNU,Clang,AppleClang>:
|
||||
-O3
|
||||
-march=native
|
||||
-fno-omit-frame-pointer
|
||||
>
|
||||
$<$<CXX_COMPILER_ID:MSVC>:
|
||||
/O2
|
||||
/GL # whole-program optimisation
|
||||
/GL
|
||||
>
|
||||
)
|
||||
|
|
@ -1,144 +0,0 @@
|
|||
# `gsl::span` vs `std::span` Benchmark
|
||||
|
||||
Performance parity tracking between `gsl::span` and `std::span` across all supported compilers, platforms, and C++ standards — as part of [microsoft/GSL#1167](https://github.com/microsoft/GSL/issues/1167) and [microsoft/GSL#1165](https://github.com/microsoft/GSL/issues/1165).
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
`gsl::span` should be a zero-overhead abstraction over `std::span`. This benchmark suite verifies that claim continuously — on every PR — so performance regressions are caught before they land in main.
|
||||
|
||||
The comparison strategy is **in-run ratio** (`gsl_ns / std_ns`): both spans are measured in the same process on the same machine at the same moment, so runner noise cancels out. A ratio close to `1.0` means parity. If `gsl::span` is more than **15% slower** than `std::span` on any benchmark, CI flags it and posts a detailed table in the PR comment.
|
||||
|
||||
---
|
||||
|
||||
## Benchmarks
|
||||
|
||||
All benchmarks run on a sorted vector of 1000 integers.
|
||||
|
||||
| Benchmark | What it tests |
|
||||
|---|---|
|
||||
| `IsSorted` | `std::is_sorted` via span iterators |
|
||||
| `IsSortedRanges` | `std::ranges::is_sorted` via the span range interface |
|
||||
| `IsSortedCustom` | Custom hand-rolled `is_sorted` loop via span iterators |
|
||||
| `MinElementAlgorithm` | `std::min_element` via span iterators |
|
||||
| `MinElementRangeFor` | Range-for loop with a custom min accumulator |
|
||||
|
||||
Each benchmark has a `StdSpan` and `GslSpan` variant. The Python script pairs them by name and computes the ratio.
|
||||
|
||||
---
|
||||
|
||||
## CI Matrix
|
||||
|
||||
The benchmark runs on every pull request across 13 configurations:
|
||||
|
||||
| OS | Compiler | C++ Standard |
|
||||
|---|---|---|
|
||||
| ubuntu-latest | GCC 13 | C++20 |
|
||||
| ubuntu-latest | GCC 14 | C++20, C++23 |
|
||||
| ubuntu-latest | Clang 17 | C++20 |
|
||||
| ubuntu-latest | Clang 18 | C++20, C++23 |
|
||||
| windows-latest | MSVC 2022 | C++20, C++23 |
|
||||
| windows-latest | clang-cl (VS 2022 bundled) | C++20, C++23 |
|
||||
| macos-14 (Apple Silicon) | Apple Clang (Xcode latest) | C++20, C++23 |
|
||||
|
||||
Results from all 13 jobs are collected and posted as a **single PR comment**, updated on every push.
|
||||
|
||||
---
|
||||
|
||||
## Repository Layout
|
||||
|
||||
```
|
||||
benchmark/
|
||||
├── CMakeLists.txt # self-contained build — fetches benchmark + googletest
|
||||
├── span_bench.cpp # the benchmark source
|
||||
├── check_regression.py # parses JSON results, writes the PR comment markdown
|
||||
└── README.md # this file
|
||||
|
||||
.github/workflows/
|
||||
└── span_benchmark.yml # CI workflow
|
||||
```
|
||||
|
||||
The benchmark folder is self-contained. `CMakeLists.txt` fetches `google/benchmark` (v1.9.0) and `google/googletest` via `FetchContent`. GSL itself is sourced from the **local repo checkout** — not a pinned tag — so the benchmark always tests the code in the PR, not a release.
|
||||
|
||||
---
|
||||
|
||||
## Running Locally
|
||||
|
||||
**Prerequisites:** CMake ≥ 3.20, a C++20-capable compiler, internet access for `FetchContent`.
|
||||
|
||||
```bash
|
||||
# From the benchmark/ directory:
|
||||
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build --target span_bench
|
||||
|
||||
# Run with JSON output (matches what CI does):
|
||||
./build/span_bench \
|
||||
--benchmark_format=json \
|
||||
--benchmark_repetitions=10 \
|
||||
--benchmark_report_aggregates_only=true \
|
||||
--benchmark_out=results.json
|
||||
|
||||
# Generate the regression report locally:
|
||||
python3 check_regression.py --threshold 0.15 results.json
|
||||
```
|
||||
|
||||
To test a specific compiler or standard:
|
||||
|
||||
```bash
|
||||
cmake -S . -B build \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_CXX_COMPILER=clang++-18 \
|
||||
-DCMAKE_CXX_STANDARD=23
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Regression Detection
|
||||
|
||||
The `check_regression.py` script:
|
||||
|
||||
1. Reads one or more Google Benchmark JSON files (one per CI matrix config)
|
||||
2. Pairs `*StdSpan` benchmarks with their `*GslSpan` counterparts by name
|
||||
3. Computes `ratio = gsl_mean / std_mean` using the 10-repetition mean
|
||||
4. Flags any ratio above `1 + threshold` (default **15%**) as a regression
|
||||
5. Writes a Markdown table per config, collected into a single PR comment
|
||||
6. Exits with code `1` if any regression is found — failing the CI check
|
||||
|
||||
```
|
||||
python3 check_regression.py [--threshold 0.15] [--output report.md] results_*.json
|
||||
```
|
||||
|
||||
### Example PR comment output
|
||||
|
||||
```
|
||||
## 📊 gsl::span vs std::span benchmark results
|
||||
|
||||
### `GCC-14-cpp20`
|
||||
| Benchmark | std mean | std σ | gsl mean | gsl σ | ratio | status |
|
||||
|----------------------|----------|--------|----------|--------|-------|----------|
|
||||
| IsSorted | 124.1 ns | ±1.2% | 125.3 ns | ±1.4% | 1.01× | ✅ 1.01× |
|
||||
| IsSortedRanges | 88.4 ns | ±0.9% | 89.1 ns | ±1.1% | 1.01× | ✅ 1.01× |
|
||||
| IsSortedCustom | 112.6 ns | ±1.5% | 113.2 ns | ±1.3% | 1.01× | ✅ 1.01× |
|
||||
| MinElementAlgorithm | 95.2 ns | ±1.0% | 96.0 ns | ±1.2% | 1.01× | ✅ 1.01× |
|
||||
| MinElementRangeFor | 98.7 ns | ±1.1% | 99.4 ns | ±0.8% | 1.01× | ✅ 1.01× |
|
||||
```
|
||||
|
||||
Status icons:
|
||||
- ✅ — within the threshold (parity)
|
||||
- 🔴 — `gsl::span` is more than 15% slower (regression, CI fails)
|
||||
- 🟢 — `gsl::span` is more than 15% faster (improvement, noted but not a failure)
|
||||
|
||||
---
|
||||
|
||||
## Noise Considerations
|
||||
|
||||
GitHub-hosted runners are shared VMs with a typical noise floor of **±10–15%** on absolute timing. The ratio strategy mitigates this because both span variants experience the same CPU conditions simultaneously. The 15% threshold is chosen to sit just above the noise floor — tight enough to catch real regressions, loose enough to avoid false positives on every PR.
|
||||
|
||||
To further reduce variance, each benchmark runs **10 repetitions** and the script uses the **mean** (not a single sample) for the ratio calculation. The stddev column in the comment table lets reviewers eyeball how stable each measurement was.
|
||||
|
||||
---
|
||||
|
||||
## Background
|
||||
|
||||
This benchmark was created in response to [microsoft/GSL#1167](https://github.com/microsoft/GSL/issues/1167), which highlighted the need to maintain performance parity with `std::span`, especially when [Safe Buffers / `-fbounds-safety`](https://clang.llvm.org/docs/SafeBuffers.html) are enabled. The initial benchmark scaffolding was provided by @galenelias.
|
||||
|
|
@ -131,7 +131,7 @@ def fmt_stddev(stddev: float, mean: float) -> str:
|
|||
|
||||
# ─── report builder ───────────────────────────────────────────────────────────
|
||||
|
||||
def build_report(json_paths: list[str], threshold: float, workflow_url: str = None) -> tuple[str, bool]:
|
||||
def build_report(json_paths: list[str], threshold: float) -> tuple[str, bool]:
|
||||
"""
|
||||
Returns (markdown_text, had_regression).
|
||||
"""
|
||||
|
|
@ -202,6 +202,7 @@ def build_report(json_paths: list[str], threshold: float, workflow_url: str = No
|
|||
config_regression = False
|
||||
for p in pairs:
|
||||
ratio, status = ratio_and_status(p["gsl_mean"], p["std_mean"], threshold)
|
||||
ratio_str = "—" if ratio is None else f"{ratio:.2f}×"
|
||||
if "regression" in status:
|
||||
had_regression = True
|
||||
config_regression = True
|
||||
|
|
@ -212,7 +213,7 @@ def build_report(json_paths: list[str], threshold: float, workflow_url: str = No
|
|||
f"| {fmt_stddev(p['std_stddev'], p['std_mean'])} "
|
||||
f"| {fmt(p['gsl_mean'])} "
|
||||
f"| {fmt_stddev(p['gsl_stddev'], p['gsl_mean'])} "
|
||||
f"| {ratio:.2f}× "
|
||||
f"| {ratio_str} "
|
||||
f"| {status} |"
|
||||
)
|
||||
|
||||
|
|
@ -229,14 +230,6 @@ def build_report(json_paths: list[str], threshold: float, workflow_url: str = No
|
|||
if not found_any:
|
||||
lines.append("> ❌ No benchmark results could be loaded.")
|
||||
|
||||
# Footer
|
||||
lines.append("---")
|
||||
lines.append(
|
||||
"_Ratio = `gsl_ns / std_ns`. "
|
||||
"Values close to 1.0 mean performance parity. "
|
||||
f"Run by [span-benchmark CI]({workflow_url})._"
|
||||
)
|
||||
|
||||
return "\n".join(lines), had_regression
|
||||
|
||||
|
||||
|
|
@ -264,15 +257,9 @@ def main():
|
|||
metavar="FILE",
|
||||
help="Write Markdown report to FILE instead of stdout.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--workflow-url",
|
||||
default=None,
|
||||
metavar="URL",
|
||||
help="GitHub URL to the workflow YAML (shows in the report footer).",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
report, had_regression = build_report(args.json_files, args.threshold, args.workflow_url)
|
||||
report, had_regression = build_report(args.json_files, args.threshold)
|
||||
|
||||
if args.output:
|
||||
Path(args.output).write_text(report, encoding="utf-8")
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
// Setup:
|
||||
// 1) cmake . -DCMAKE_BUILD_TYPE=Release
|
||||
// 2) cmake --build . --config Release
|
||||
|
||||
#include "gsl/span"
|
||||
#include <algorithm>
|
||||
#include <benchmark/benchmark.h>
|
||||
#include <numeric>
|
||||
#include <limits>
|
||||
#include <ranges>
|
||||
#include <span>
|
||||
#include <vector>
|
||||
|
||||
static std::vector<int> make_vector()
|
||||
{
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue