ament_cmake_google_benchmark

This package contains logic for invoking Google Benchmark tests and is comprised of three main parts:

1. The ament_add_google_benchmark* CMake macros.
   These macros are used in ament packages to compile Google Benchmark executables, run Google Benchmark tests with CTest, or both.
2. The run_and_convert.py test wrapper.
   This script is used to convert the Google Benchmark JSON results to a custom format which can be consumed by the Jenkins Benchmark plugin. The script also combines the generated results with elements from an overlay file, which may contain additional context for the benchmarks, such as descriptions or value thresholds.
3. The benchmark_schema.json output schema.
   This schema describes the output format for the run_and_convert.py test wrapper, and should be used by Jenkins in parsing the results for aggregation and threshold checks. It is also the schema used by the overlay file.

Examples

The source file format is well-described in the Google Benchmark README. Here is a simple example:

#include <benchmark/benchmark.h>

static void increment_perf(benchmark::State & state)
{
  size_t i = 0;
  for (auto _ : state) {
    i++;
  }
}
BENCHMARK(increment_perf);

To compile and run this benchmark, the following is added to the CMakeLists.txt:

if(BUILD_TESTING)
  find_package(ament_cmake_google_benchmark REQUIRED)
  ament_add_google_benchmark(simple_benchmark test/simple_benchmark.cpp)
endif()

Jenkins can also trigger a build warning when the metrics exceed predefined thresholds. Here is an example overlay file to sets a threshold of 0.6% deviation above or below the average metric from all previous Jenkins job builds which include that metric:

{
  "package_name.simple_benchmark": {
    "increment_perf": {
      "real_time": {
        "thresholds": [
          {
            "method": "percentageaverage",
            "percentage": 0.6
          }
        ]
      }
    }
  }
}

This overlay file can be generated at build time or checked in as a static file. The location of the file is communicated to this package using the AMENT_CMAKE_GOOGLE_BENCHMARK_OVERLAY CMake variable. For example:

if(BUILD_TESTING)
  set(AMENT_CMAKE_GOOGLE_BENCHMARK_OVERLAY test/benchmark_thresholds.json)
  ...
endif()

Changelog for package ament_cmake_google_benchmark

2.7.2 (2024-11-19)

2.7.1 (2024-10-03)

2.7.0 (2024-06-25)

2.6.1 (2024-06-17)

2.6.0 (2024-04-26)

2.5.0 (2024-04-16)

2.4.0 (2024-03-28)

• Update maintainer list in package.xml files (#503)
• Contributors: Michael Jeronimo

2.3.2 (2023-12-26)

2.3.1 (2023-11-06)

2.3.0 (2023-09-07)

2.2.2 (2023-08-21)

2.2.1 (2023-06-21)

2.2.0 (2023-06-07)

2.1.0 (2023-04-26)

2.0.2 (2023-04-12)

2.0.1 (2023-04-11)

2.0.0 (2023-04-11)

1.5.3 (2023-02-13)

• [rolling] Update maintainers - 2022-11-07 (#411)
  • Update maintainers to Michael Jeronimo
• Contributors: Audrow Nash

1.5.2 (2022-11-02)

1.5.1 (2022-09-13)

1.5.0 (2022-07-11)

1.4.0 (2022-04-29)

1.3.1 (2022-03-28)

1.3.0 (2022-02-17)

• Update forthcoming version in changelog
• Contributors: Audrow Nash

1.2.1 (2022-01-14)

• Resolve various ament_lint linter violations (#360) We can't add ament_lint linters in ament_cmake in the traditional way without creating a circular dependency between the repositories. Even though we can't automatically enforce linting, it's still a good idea to try to keep conformance where possible.
• Update maintainers to Michael Jeronimo and Michel Hidalgo (#362)
• Contributors: Audrow Nash, Scott K Logan

1.2.0 (2021-10-29)

• Use FindPython3 instead of FindPythonInterp (#355)
• Update maintainers (#336)
• Contributors: Chris Lalancette, Shane Loretz

1.1.4 (2021-05-06)

1.1.3 (2021-03-09)

1.1.2 (2021-02-26 22:59)

1.1.1 (2021-02-26 19:12)

1.1.0 (2021-02-24)

• Serialize benchmarks within CTest by default (#308)
• Contributors: Scott K Logan

1.0.4 (2021-01-25)

1.0.3 (2020-12-10)

1.0.2 (2020-12-07)

• Handle runtime failures in Google Benchmark (#294) This change will handle runtime failures in Google Benchmark by propagating error information from Google Benchmark to both CTest and the Jenkins benchmark plugin.
• Use consistent string format and resolve flake8 (#295) Follow-up to a5fb3112b5c46c42b1824c96af4171d469eb13bf
• Make ament_cmake_test a dep of ament_cmake_google_benchmark (#293)
• Catch JSONDecodeError and printout some debug info (#291)
• Update package maintainers. (#286)
• Contributors: Michel Hidalgo, Scott K Logan, brawner

1.0.1 (2020-09-10)

• Make AMENT_RUN_PERFORMANCE_TESTS a CMake option (#280)
• Skip performance tests using a CMake variable (#278) These tests can be fairly heavy, so we don't want to run them by default. It would be better if there was a way to skip the tests by default in such a way that they could be specifically un-skipped at runtime, but I can't find a mechanism in CMake or CTest that would allow us to achieve that behavior without leveraging environment variables.
• Handle Google Benchmark 'aggregate' results (#276) Previously, I assumed all results generated by Google Benchmark were of 'iteration' type. Now that I have more experience with Google Benchmark, I've started generating aggregate results, which contain some different properties. This change adds support for aggregate results and should make it easy to add any other result schemas we encounter in the future. For forward-compatibility, unsupported types will generate a warning message but will not fail the test. This makes the conversion tolerant to Google Benchmark adding new measures for existing mechanisms.
• Initial Google Benchmark results conversion (#275)
• Contributors: Scott K Logan

1.0.0 (2020-07-22)

• Handle missing results file for Google Benchmark (#265)
• Initial ament_cmake_google_benchmark package (#261)
• Contributors: Scott K Logan

0.9.6 (2020-06-23)

0.9.5 (2020-06-02)

0.9.4 (2020-05-26)

0.9.3 (2020-05-19)

0.9.2 (2020-05-07)

0.9.1 (2020-04-24 15:45)

0.9.0 (2020-04-24 12:25)

0.8.1 (2019-10-23)

0.8.0 (2019-10-04)

0.7.3 (2019-05-29)

0.7.2 (2019-05-20)

0.7.1 (2019-05-07)

0.7.0 (2019-04-08)

0.6.0 (2018-11-13)

0.5.1 (2018-07-17)

0.5.0 (2018-06-13)

0.4.0 (2017-12-08)