Package Summary

Tags No category tags.
Version 1.1.4
License Apache License 2.0
Build type AMENT_CMAKE
Use RECOMMENDED

Repository Summary

Checkout URI https://github.com/ament/ament_cmake.git
VCS Type git
VCS Version galactic
Last Updated 2021-05-06
Dev Status DEVELOPED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Package Description

The ability to add Google Benchmark tests in the ament buildsystem in CMake.

Additional Links

No additional links.

Maintainers

  • Dirk Thomas
  • Michel Hidalgo

Authors

  • Scott K Logan

ament_cmake_google_benchmark

This package contains logic for invoking Google Benchmark tests and is comprised of three main parts:

  1. The ament_add_google_benchmark* CMake macros.\ These macros are used in ament packages to compile Google Benchmark executables, run Google Benchmark tests with CTest, or both.
  2. The run_and_convert.py test wrapper.\ This script is used to convert the Google Benchmark JSON results to a custom format which can be consumed by the Jenkins Benchmark plugin. The script also combines the generated results with elements from an overlay file, which may contain additional context for the benchmarks, such as descriptions or value thresholds.
  3. The benchmark_schema.json output schema.\ This schema describes the output format for the run_and_convert.py test wrapper, and should be used by Jenkins in parsing the results for aggregation and threshold checks. It is also the schema used by the overlay file.

Examples

The source file format is well-described in the Google Benchmark README. Here is a simple example:

#include <benchmark/benchmark.h>

static void increment_perf(benchmark::State & state)
{
  size_t i = 0;
  for (auto _ : state) {
    i++;
  }
}
BENCHMARK(increment_perf);

To compile and run this benchmark, the following is added to the CMakeLists.txt:

if(BUILD_TESTING)
  find_package(ament_cmake_google_benchmark REQUIRED)
  ament_add_google_benchmark(simple_benchmark test/simple_benchmark.cpp)
endif()

Jenkins can also trigger a build warning when the metrics exceed predefined thresholds. Here is an example overlay file to sets a threshold of 0.6% deviation above or below the average metric from all previous Jenkins job builds which include that metric:

{
  "package_name.simple_benchmark": {
    "increment_perf": {
      "real_time": {
        "thresholds": [
          {
            "method": "percentageaverage",
            "percentage": 0.6
          }
        ]
      }
    }
  }
}

This overlay file can be generated at build time or checked in as a static file. The location of the file is communicated to this package using the AMENT_CMAKE_GOOGLE_BENCHMARK_OVERLAY CMake variable. For example:

if(BUILD_TESTING)
  set(AMENT_CMAKE_GOOGLE_BENCHMARK_OVERLAY test/benchmark_thresholds.json)
  ...
endif()

CHANGELOG

Changelog for package ament_cmake_google_benchmark

1.1.4 (2021-05-06)

1.1.3 (2021-03-09)

1.1.2 (2021-02-26 22:59)

1.1.1 (2021-02-26 19:12)

1.1.0 (2021-02-24)

  • Serialize benchmarks within CTest by default (#308)
  • Contributors: Scott K Logan

1.0.4 (2021-01-25)

1.0.3 (2020-12-10)

1.0.2 (2020-12-07)

  • Handle runtime failures in Google Benchmark (#294) This change will handle runtime failures in Google Benchmark by propagating error information from Google Benchmark to both CTest and the Jenkins benchmark plugin.
  • Use consistent string format and resolve flake8 (#295) Follow-up to a5fb3112b5c46c42b1824c96af4171d469eb13bf
  • Make ament_cmake_test a dep of ament_cmake_google_benchmark (#293)
  • Catch JSONDecodeError and printout some debug info (#291)
  • Update package maintainers. (#286)
  • Contributors: Michel Hidalgo, Scott K Logan, brawner

1.0.1 (2020-09-10)

  • Make AMENT_RUN_PERFORMANCE_TESTS a CMake option (#280)
  • Skip performance tests using a CMake variable (#278) These tests can be fairly heavy, so we don\'t want to run them by default. It would be better if there was a way to skip the tests by default in such a way that they could be specifically un-skipped at runtime, but I can\'t find a mechanism in CMake or CTest that would allow us to achieve that behavior without leveraging environment variables.
  • Handle Google Benchmark \'aggregate\' results (#276) Previously, I assumed all results generated by Google Benchmark were of \'iteration\' type. Now that I have more experience with Google Benchmark, I\'ve started generating aggregate results, which contain some different properties. This change adds support for aggregate results and should make it easy to add any other result schemas we encounter in the future. For forward-compatibility, unsupported types will generate a warning message but will not fail the test. This makes the conversion tolerant to Google Benchmark adding new measures for existing mechanisms.
  • Initial Google Benchmark results conversion (#275)
  • Contributors: Scott K Logan

1.0.0 (2020-07-22)

  • Handle missing results file for Google Benchmark (#265)
  • Initial ament_cmake_google_benchmark package (#261)
  • Contributors: Scott K Logan

0.9.6 (2020-06-23)

0.9.5 (2020-06-02)

0.9.4 (2020-05-26)

0.9.3 (2020-05-19)

0.9.2 (2020-05-07)

0.9.1 (2020-04-24 15:45)

0.9.0 (2020-04-24 12:25)

0.8.1 (2019-10-23)

0.8.0 (2019-10-04)

0.7.3 (2019-05-29)

0.7.2 (2019-05-20)

0.7.1 (2019-05-07)

0.7.0 (2019-04-08)

0.6.0 (2018-11-13)

0.5.1 (2018-07-17)

0.5.0 (2018-06-13)

0.4.0 (2017-12-08)

Wiki Tutorials

See ROS Wiki Tutorials for more details.

Source Tutorials

Not currently indexed.

Launch files

No launch files found

Messages

No message files found.

Services

No service files found

Plugins

No plugins found.

Recent questions tagged ament_cmake_google_benchmark at answers.ros.org

Package Summary

Tags No category tags.
Version 0.9.9
License Apache License 2.0
Build type AMENT_CMAKE
Use RECOMMENDED

Repository Summary

Checkout URI https://github.com/ament/ament_cmake.git
VCS Type git
VCS Version foxy
Last Updated 2021-08-31
Dev Status DEVELOPED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Package Description

The ability to add Google Benchmark tests in the ament buildsystem in CMake.

Additional Links

No additional links.

Maintainers

  • Scott K Logan

Authors

No additional authors.

ament_cmake_google_benchmark

This package contains logic for invoking Google Benchmark tests and is comprised of three main parts:

  1. The ament_add_google_benchmark* CMake macros.\ These macros are used in ament packages to compile Google Benchmark executables, run Google Benchmark tests with CTest, or both.
  2. The run_and_convert.py test wrapper.\ This script is used to convert the Google Benchmark JSON results to a custom format which can be consumed by the Jenkins Benchmark plugin. The script also combines the generated results with elements from an overlay file, which may contain additional context for the benchmarks, such as descriptions or value thresholds.
  3. The benchmark_schema.json output schema.\ This schema describes the output format for the run_and_convert.py test wrapper, and should be used by Jenkins in parsing the results for aggregation and threshold checks. It is also the schema used by the overlay file.

Examples

The source file format is well-described in the Google Benchmark README. Here is a simple example:

#include <benchmark/benchmark.h>

static void increment_perf(benchmark::State & state)
{
  size_t i = 0;
  for (auto _ : state) {
    i++;
  }
}
BENCHMARK(increment_perf);

To compile and run this benchmark, the following is added to the CMakeLists.txt:

if(BUILD_TESTING)
  find_package(ament_cmake_google_benchmark REQUIRED)
  ament_add_google_benchmark(simple_benchmark test/simple_benchmark.cpp)
endif()

Jenkins can also trigger a build warning when the metrics exceed predefined thresholds. Here is an example overlay file to sets a threshold of 0.6% deviation above or below the average metric from all previous Jenkins job builds which include that metric:

{
  "package_name.simple_benchmark": {
    "increment_perf": {
      "real_time": {
        "thresholds": [
          {
            "method": "percentageaverage",
            "percentage": 0.6
          }
        ]
      }
    }
  }
}

This overlay file can be generated at build time or checked in as a static file. The location of the file is communicated to this package using the AMENT_CMAKE_GOOGLE_BENCHMARK_OVERLAY CMake variable. For example:

if(BUILD_TESTING)
  set(AMENT_CMAKE_GOOGLE_BENCHMARK_OVERLAY test/benchmark_thresholds.json)
  ...
endif()

CHANGELOG
No CHANGELOG found.

Wiki Tutorials

See ROS Wiki Tutorials for more details.

Source Tutorials

Not currently indexed.

Launch files

No launch files found

Messages

No message files found.

Services

No service files found

Plugins

No plugins found.

Recent questions tagged ament_cmake_google_benchmark at answers.ros.org

Package Summary

Tags No category tags.
Version 1.1.4
License Apache License 2.0
Build type AMENT_CMAKE
Use RECOMMENDED

Repository Summary

Checkout URI https://github.com/ament/ament_cmake.git
VCS Type git
VCS Version master
Last Updated 2021-10-13
Dev Status DEVELOPED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Package Description

The ability to add Google Benchmark tests in the ament buildsystem in CMake.

Additional Links

No additional links.

Maintainers

  • Michel Hidalgo

Authors

  • Scott K Logan
  • Dirk Thomas

ament_cmake_google_benchmark

This package contains logic for invoking Google Benchmark tests and is comprised of three main parts:

  1. The ament_add_google_benchmark* CMake macros.\ These macros are used in ament packages to compile Google Benchmark executables, run Google Benchmark tests with CTest, or both.
  2. The run_and_convert.py test wrapper.\ This script is used to convert the Google Benchmark JSON results to a custom format which can be consumed by the Jenkins Benchmark plugin. The script also combines the generated results with elements from an overlay file, which may contain additional context for the benchmarks, such as descriptions or value thresholds.
  3. The benchmark_schema.json output schema.\ This schema describes the output format for the run_and_convert.py test wrapper, and should be used by Jenkins in parsing the results for aggregation and threshold checks. It is also the schema used by the overlay file.

Examples

The source file format is well-described in the Google Benchmark README. Here is a simple example:

#include <benchmark/benchmark.h>

static void increment_perf(benchmark::State & state)
{
  size_t i = 0;
  for (auto _ : state) {
    i++;
  }
}
BENCHMARK(increment_perf);

To compile and run this benchmark, the following is added to the CMakeLists.txt:

if(BUILD_TESTING)
  find_package(ament_cmake_google_benchmark REQUIRED)
  ament_add_google_benchmark(simple_benchmark test/simple_benchmark.cpp)
endif()

Jenkins can also trigger a build warning when the metrics exceed predefined thresholds. Here is an example overlay file to sets a threshold of 0.6% deviation above or below the average metric from all previous Jenkins job builds which include that metric:

{
  "package_name.simple_benchmark": {
    "increment_perf": {
      "real_time": {
        "thresholds": [
          {
            "method": "percentageaverage",
            "percentage": 0.6
          }
        ]
      }
    }
  }
}

This overlay file can be generated at build time or checked in as a static file. The location of the file is communicated to this package using the AMENT_CMAKE_GOOGLE_BENCHMARK_OVERLAY CMake variable. For example:

if(BUILD_TESTING)
  set(AMENT_CMAKE_GOOGLE_BENCHMARK_OVERLAY test/benchmark_thresholds.json)
  ...
endif()

CHANGELOG

Changelog for package ament_cmake_google_benchmark

1.1.4 (2021-05-06)

1.1.3 (2021-03-09)

1.1.2 (2021-02-26 22:59)

1.1.1 (2021-02-26 19:12)

1.1.0 (2021-02-24)

  • Serialize benchmarks within CTest by default (#308)
  • Contributors: Scott K Logan

1.0.4 (2021-01-25)

1.0.3 (2020-12-10)

1.0.2 (2020-12-07)

  • Handle runtime failures in Google Benchmark (#294) This change will handle runtime failures in Google Benchmark by propagating error information from Google Benchmark to both CTest and the Jenkins benchmark plugin.
  • Use consistent string format and resolve flake8 (#295) Follow-up to a5fb3112b5c46c42b1824c96af4171d469eb13bf
  • Make ament_cmake_test a dep of ament_cmake_google_benchmark (#293)
  • Catch JSONDecodeError and printout some debug info (#291)
  • Update package maintainers. (#286)
  • Contributors: Michel Hidalgo, Scott K Logan, brawner

1.0.1 (2020-09-10)

  • Make AMENT_RUN_PERFORMANCE_TESTS a CMake option (#280)
  • Skip performance tests using a CMake variable (#278) These tests can be fairly heavy, so we don\'t want to run them by default. It would be better if there was a way to skip the tests by default in such a way that they could be specifically un-skipped at runtime, but I can\'t find a mechanism in CMake or CTest that would allow us to achieve that behavior without leveraging environment variables.
  • Handle Google Benchmark \'aggregate\' results (#276) Previously, I assumed all results generated by Google Benchmark were of \'iteration\' type. Now that I have more experience with Google Benchmark, I\'ve started generating aggregate results, which contain some different properties. This change adds support for aggregate results and should make it easy to add any other result schemas we encounter in the future. For forward-compatibility, unsupported types will generate a warning message but will not fail the test. This makes the conversion tolerant to Google Benchmark adding new measures for existing mechanisms.
  • Initial Google Benchmark results conversion (#275)
  • Contributors: Scott K Logan

1.0.0 (2020-07-22)

  • Handle missing results file for Google Benchmark (#265)
  • Initial ament_cmake_google_benchmark package (#261)
  • Contributors: Scott K Logan

0.9.6 (2020-06-23)

0.9.5 (2020-06-02)

0.9.4 (2020-05-26)

0.9.3 (2020-05-19)

0.9.2 (2020-05-07)

0.9.1 (2020-04-24 15:45)

0.9.0 (2020-04-24 12:25)

0.8.1 (2019-10-23)

0.8.0 (2019-10-04)

0.7.3 (2019-05-29)

0.7.2 (2019-05-20)

0.7.1 (2019-05-07)

0.7.0 (2019-04-08)

0.6.0 (2018-11-13)

0.5.1 (2018-07-17)

0.5.0 (2018-06-13)

0.4.0 (2017-12-08)

Wiki Tutorials

See ROS Wiki Tutorials for more details.

Source Tutorials

Not currently indexed.

Launch files

No launch files found

Messages

No message files found.

Services

No service files found

Plugins

No plugins found.

Recent questions tagged ament_cmake_google_benchmark at answers.ros.org

No version for distro noetic. Known supported distros are highlighted in the buttons above.
No version for distro melodic. Known supported distros are highlighted in the buttons above.
No version for distro ardent. Known supported distros are highlighted in the buttons above.
No version for distro bouncy. Known supported distros are highlighted in the buttons above.
No version for distro crystal. Known supported distros are highlighted in the buttons above.
No version for distro eloquent. Known supported distros are highlighted in the buttons above.
No version for distro dashing. Known supported distros are highlighted in the buttons above.
No version for distro lunar. Known supported distros are highlighted in the buttons above.
No version for distro jade. Known supported distros are highlighted in the buttons above.
No version for distro indigo. Known supported distros are highlighted in the buttons above.
No version for distro hydro. Known supported distros are highlighted in the buttons above.
No version for distro kinetic. Known supported distros are highlighted in the buttons above.