Package Summary

Tags No category tags.
Version 1.3.5
License Apache 2.0
Build type AMENT_PYTHON
Use RECOMMENDED

Repository Summary

Checkout URI https://gitlab.com/ApexAI/performance_test.git
VCS Type git
VCS Version master
Last Updated 2022-12-05
Dev Status MAINTAINED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Package Description

Apex.AI performance_test runner, plotter, and reporter

Additional Links

No additional links.

Maintainers

  • Apex AI, Inc.

Authors

No additional authors.

performance_report

This package serves two purposes:

  1. Run multiple performance_test experiments
  2. Plot the combined results of those experiments

Quick start

Install the required dependencies

# install bokeh and selenium
python3 -m pip install bokeh selenium pandas
# install firefox-geckodriver
sudo apt install firefox-geckodriver

Note: all the commands below are ran from the colcon_ws where performance_test/performance_report is installed.

# Build performance_test and performance_report
colcon build

# Set up the environment
source install/setup.bash

# Run perf_test for each experiment in the yaml file
ros2 run performance_report runner \
  --log-dir perf_logs \
  --test-name experiments \
  --configs src/performance_test/performance_report/cfg/runner/run_one_experiment.yaml

# The runner generates log files to the specified directory: `./perf_logs/experiements/`

# Generate the plots configured in the specified yaml file
ros2 run performance_report plotter \
  --log-dir perf_logs \
  --configs src/performance_test/performance_report/cfg/plotter/plot_one_experiment.yaml

# The generated plots will be saved in `./perf_logs`

# Generate the reports configured in the specified yaml file
ros2 run performance_report reporter \
  --log-dir perf_logs \
  --configs src/performance_test/performance_report/cfg/reporter/report_one_experiment.yaml

The above example only runs and plots one experiement, however multiple experiements can be ran automatically by specifying it in the configuration file.

Try the above commands again, this time replacing the yaml file with *_many_experiements.yaml for each executable.

When running many experiments the runner will by default skip any experiments that already have log files generated in the specified log_dir. This can be overridden by adding the -f or --force argument to the command.

Running the same experiments on multiple platforms

Suppose you want to run an experiment on multiple platforms, then combine the results into a single report. First, pass the --test-name arg to runner, to differentiate the result sets:

# on platform 1:
ros2 run performance_report runner --test_name platform1 -l log_dir -c run.yaml
# results will be stored in ./log_dir/platform1/

# on platform 2:
ros2 run performance_report runner --test_name platform2 -l log_dir -c run.yaml
# results will be stored in ./log_dir/platform2/

You can then combine these results into a single log_dir, on the platform where you will run plotter or reporter. Then, in your plotter or reporter configuration file, set test_name in each dataset, to select results from that platform's result set:

# report.yaml
datasets:
  dataset_p1:
    test_name: platform1  # this matches the --test-name passed to runner
    # other fields...
  dataset_p2:
    test_name: platform2  # this matches the --test-name passed to runner
    # other fields...
reports:
  # ...

ros2 run performance_report reporter -l log_dir -c report.yaml

Notes

  • Currently, this tool is intended for ROS 2 with rmw_cyclone_dds, or Apex.OS with Apex.Middleware. It has not been tested with any other transport.
  • If the run configuration includes SHMEM or ZERO_COPY transport, then a file for configuring the middleware will be created to enable the shared memory transfer.
    • You must start RouDi before running the experiments. This tool will not automatically start it for you.
CHANGELOG

Changelog for package performance_report

X.Y.Z (YYYY/MM/DD)

1.3.5 (2022/12/05)

1.3.4 (2022/11/28)

1.3.3 (2022/11/28)

Fixed

  • Do not try to create a box-and-whisker for a file that contains no measurements

1.3.2 (2022/11/21)

1.3.1 (2022/11/21)

1.3.0 (2022/08/25)

Added

  • The reporter configuration supports box-and-whisker latency plots:
    • set the x_range to Experiment
    • set the y_range to latency_mean
    • set datasets to one or more datasets, each containing a single experiment
    • an example can be found in cfg/reporter/report_many_experiments.yaml ### Changed
  • Expanded the transport setting into the following two settings:
    • process_configuration:
      • INTRA_PROCESS
      • INTER_PROCESS
    • sample_transport:
      • BY_COPY
      • SHARED_MEMORY
      • LOANED_SAMPLES

1.2.1 (2022/06/30)

1.2.0 (2022/06/28)

Changed

  • In the reporter configuration, the template_name value may be an array

1.1.2 (2022/06/08)

1.1.1 (2022/06/07)

Fixed

  • Bokeh line style can be specified in the plotter and reporter .yaml files

1.1.0 (2022/06/02)

Fixed

  • Fix the GBP builds by removing python3-bokeh-pip from package.xml

1.0.0 (2022/05/12)

Added

  • Shared memory experiments are now compatible with both Apex.Middleware and rmw_cyclonedds_cpp
  • commander tool to emit the commands for running the experiments, instead of running them directly ### Changed
  • Use the new perf_test CLI args for QOS settings instead of old flags ### Deprecated ### Removed ### Fixed

Wiki Tutorials

See ROS Wiki Tutorials for more details.

Source Tutorials

Not currently indexed.

Launch files

No launch files found

Messages

No message files found.

Services

No service files found

Plugins

No plugins found.

Recent questions tagged performance_report at answers.ros.org

Package Summary

Tags No category tags.
Version 1.3.5
License Apache 2.0
Build type AMENT_PYTHON
Use RECOMMENDED

Repository Summary

Checkout URI https://gitlab.com/ApexAI/performance_test.git
VCS Type git
VCS Version master
Last Updated 2022-12-05
Dev Status MAINTAINED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Package Description

Apex.AI performance_test runner, plotter, and reporter

Additional Links

No additional links.

Maintainers

  • Apex AI, Inc.

Authors

No additional authors.

performance_report

This package serves two purposes:

  1. Run multiple performance_test experiments
  2. Plot the combined results of those experiments

Quick start

Install the required dependencies

# install bokeh and selenium
python3 -m pip install bokeh selenium pandas
# install firefox-geckodriver
sudo apt install firefox-geckodriver

Note: all the commands below are ran from the colcon_ws where performance_test/performance_report is installed.

# Build performance_test and performance_report
colcon build

# Set up the environment
source install/setup.bash

# Run perf_test for each experiment in the yaml file
ros2 run performance_report runner \
  --log-dir perf_logs \
  --test-name experiments \
  --configs src/performance_test/performance_report/cfg/runner/run_one_experiment.yaml

# The runner generates log files to the specified directory: `./perf_logs/experiements/`

# Generate the plots configured in the specified yaml file
ros2 run performance_report plotter \
  --log-dir perf_logs \
  --configs src/performance_test/performance_report/cfg/plotter/plot_one_experiment.yaml

# The generated plots will be saved in `./perf_logs`

# Generate the reports configured in the specified yaml file
ros2 run performance_report reporter \
  --log-dir perf_logs \
  --configs src/performance_test/performance_report/cfg/reporter/report_one_experiment.yaml

The above example only runs and plots one experiement, however multiple experiements can be ran automatically by specifying it in the configuration file.

Try the above commands again, this time replacing the yaml file with *_many_experiements.yaml for each executable.

When running many experiments the runner will by default skip any experiments that already have log files generated in the specified log_dir. This can be overridden by adding the -f or --force argument to the command.

Running the same experiments on multiple platforms

Suppose you want to run an experiment on multiple platforms, then combine the results into a single report. First, pass the --test-name arg to runner, to differentiate the result sets:

# on platform 1:
ros2 run performance_report runner --test_name platform1 -l log_dir -c run.yaml
# results will be stored in ./log_dir/platform1/

# on platform 2:
ros2 run performance_report runner --test_name platform2 -l log_dir -c run.yaml
# results will be stored in ./log_dir/platform2/

You can then combine these results into a single log_dir, on the platform where you will run plotter or reporter. Then, in your plotter or reporter configuration file, set test_name in each dataset, to select results from that platform's result set:

# report.yaml
datasets:
  dataset_p1:
    test_name: platform1  # this matches the --test-name passed to runner
    # other fields...
  dataset_p2:
    test_name: platform2  # this matches the --test-name passed to runner
    # other fields...
reports:
  # ...

ros2 run performance_report reporter -l log_dir -c report.yaml

Notes

  • Currently, this tool is intended for ROS 2 with rmw_cyclone_dds, or Apex.OS with Apex.Middleware. It has not been tested with any other transport.
  • If the run configuration includes SHMEM or ZERO_COPY transport, then a file for configuring the middleware will be created to enable the shared memory transfer.
    • You must start RouDi before running the experiments. This tool will not automatically start it for you.
CHANGELOG

Changelog for package performance_report

X.Y.Z (YYYY/MM/DD)

1.3.5 (2022/12/05)

1.3.4 (2022/11/28)

1.3.3 (2022/11/28)

Fixed

  • Do not try to create a box-and-whisker for a file that contains no measurements

1.3.2 (2022/11/21)

1.3.1 (2022/11/21)

1.3.0 (2022/08/25)

Added

  • The reporter configuration supports box-and-whisker latency plots:
    • set the x_range to Experiment
    • set the y_range to latency_mean
    • set datasets to one or more datasets, each containing a single experiment
    • an example can be found in cfg/reporter/report_many_experiments.yaml ### Changed
  • Expanded the transport setting into the following two settings:
    • process_configuration:
      • INTRA_PROCESS
      • INTER_PROCESS
    • sample_transport:
      • BY_COPY
      • SHARED_MEMORY
      • LOANED_SAMPLES

1.2.1 (2022/06/30)

1.2.0 (2022/06/28)

Changed

  • In the reporter configuration, the template_name value may be an array

1.1.2 (2022/06/08)

1.1.1 (2022/06/07)

Fixed

  • Bokeh line style can be specified in the plotter and reporter .yaml files

1.1.0 (2022/06/02)

Fixed

  • Fix the GBP builds by removing python3-bokeh-pip from package.xml

1.0.0 (2022/05/12)

Added

  • Shared memory experiments are now compatible with both Apex.Middleware and rmw_cyclonedds_cpp
  • commander tool to emit the commands for running the experiments, instead of running them directly ### Changed
  • Use the new perf_test CLI args for QOS settings instead of old flags ### Deprecated ### Removed ### Fixed

Wiki Tutorials

See ROS Wiki Tutorials for more details.

Source Tutorials

Not currently indexed.

Launch files

No launch files found

Messages

No message files found.

Services

No service files found

Plugins

No plugins found.

Recent questions tagged performance_report at answers.ros.org

Package Summary

Tags No category tags.
Version 1.3.5
License Apache 2.0
Build type AMENT_PYTHON
Use RECOMMENDED

Repository Summary

Checkout URI https://gitlab.com/ApexAI/performance_test.git
VCS Type git
VCS Version master
Last Updated 2022-12-05
Dev Status MAINTAINED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Package Description

Apex.AI performance_test runner, plotter, and reporter

Additional Links

No additional links.

Maintainers

  • Apex AI, Inc.

Authors

No additional authors.

performance_report

This package serves two purposes:

  1. Run multiple performance_test experiments
  2. Plot the combined results of those experiments

Quick start

Install the required dependencies

# install bokeh and selenium
python3 -m pip install bokeh selenium pandas
# install firefox-geckodriver
sudo apt install firefox-geckodriver

Note: all the commands below are ran from the colcon_ws where performance_test/performance_report is installed.

# Build performance_test and performance_report
colcon build

# Set up the environment
source install/setup.bash

# Run perf_test for each experiment in the yaml file
ros2 run performance_report runner \
  --log-dir perf_logs \
  --test-name experiments \
  --configs src/performance_test/performance_report/cfg/runner/run_one_experiment.yaml

# The runner generates log files to the specified directory: `./perf_logs/experiements/`

# Generate the plots configured in the specified yaml file
ros2 run performance_report plotter \
  --log-dir perf_logs \
  --configs src/performance_test/performance_report/cfg/plotter/plot_one_experiment.yaml

# The generated plots will be saved in `./perf_logs`

# Generate the reports configured in the specified yaml file
ros2 run performance_report reporter \
  --log-dir perf_logs \
  --configs src/performance_test/performance_report/cfg/reporter/report_one_experiment.yaml

The above example only runs and plots one experiement, however multiple experiements can be ran automatically by specifying it in the configuration file.

Try the above commands again, this time replacing the yaml file with *_many_experiements.yaml for each executable.

When running many experiments the runner will by default skip any experiments that already have log files generated in the specified log_dir. This can be overridden by adding the -f or --force argument to the command.

Running the same experiments on multiple platforms

Suppose you want to run an experiment on multiple platforms, then combine the results into a single report. First, pass the --test-name arg to runner, to differentiate the result sets:

# on platform 1:
ros2 run performance_report runner --test_name platform1 -l log_dir -c run.yaml
# results will be stored in ./log_dir/platform1/

# on platform 2:
ros2 run performance_report runner --test_name platform2 -l log_dir -c run.yaml
# results will be stored in ./log_dir/platform2/

You can then combine these results into a single log_dir, on the platform where you will run plotter or reporter. Then, in your plotter or reporter configuration file, set test_name in each dataset, to select results from that platform's result set:

# report.yaml
datasets:
  dataset_p1:
    test_name: platform1  # this matches the --test-name passed to runner
    # other fields...
  dataset_p2:
    test_name: platform2  # this matches the --test-name passed to runner
    # other fields...
reports:
  # ...

ros2 run performance_report reporter -l log_dir -c report.yaml

Notes

  • Currently, this tool is intended for ROS 2 with rmw_cyclone_dds, or Apex.OS with Apex.Middleware. It has not been tested with any other transport.
  • If the run configuration includes SHMEM or ZERO_COPY transport, then a file for configuring the middleware will be created to enable the shared memory transfer.
    • You must start RouDi before running the experiments. This tool will not automatically start it for you.
CHANGELOG

Changelog for package performance_report

X.Y.Z (YYYY/MM/DD)

1.3.5 (2022/12/05)

1.3.4 (2022/11/28)

1.3.3 (2022/11/28)

Fixed

  • Do not try to create a box-and-whisker for a file that contains no measurements

1.3.2 (2022/11/21)

1.3.1 (2022/11/21)

1.3.0 (2022/08/25)

Added

  • The reporter configuration supports box-and-whisker latency plots:
    • set the x_range to Experiment
    • set the y_range to latency_mean
    • set datasets to one or more datasets, each containing a single experiment
    • an example can be found in cfg/reporter/report_many_experiments.yaml ### Changed
  • Expanded the transport setting into the following two settings:
    • process_configuration:
      • INTRA_PROCESS
      • INTER_PROCESS
    • sample_transport:
      • BY_COPY
      • SHARED_MEMORY
      • LOANED_SAMPLES

1.2.1 (2022/06/30)

1.2.0 (2022/06/28)

Changed

  • In the reporter configuration, the template_name value may be an array

1.1.2 (2022/06/08)

1.1.1 (2022/06/07)

Fixed

  • Bokeh line style can be specified in the plotter and reporter .yaml files

1.1.0 (2022/06/02)

Fixed

  • Fix the GBP builds by removing python3-bokeh-pip from package.xml

1.0.0 (2022/05/12)

Added

  • Shared memory experiments are now compatible with both Apex.Middleware and rmw_cyclonedds_cpp
  • commander tool to emit the commands for running the experiments, instead of running them directly ### Changed
  • Use the new perf_test CLI args for QOS settings instead of old flags ### Deprecated ### Removed ### Fixed

Wiki Tutorials

See ROS Wiki Tutorials for more details.

Source Tutorials

Not currently indexed.

Launch files

No launch files found

Messages

No message files found.

Services

No service files found

Plugins

No plugins found.

Recent questions tagged performance_report at answers.ros.org

Package Summary

Tags No category tags.
Version 1.3.5
License Apache 2.0
Build type AMENT_PYTHON
Use RECOMMENDED

Repository Summary

Checkout URI https://gitlab.com/ApexAI/performance_test.git
VCS Type git
VCS Version master
Last Updated 2022-12-05
Dev Status MAINTAINED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Package Description

Apex.AI performance_test runner, plotter, and reporter

Additional Links

No additional links.

Maintainers

  • Apex AI, Inc.

Authors

No additional authors.

performance_report

This package serves two purposes:

  1. Run multiple performance_test experiments
  2. Plot the combined results of those experiments

Quick start

Install the required dependencies

# install bokeh and selenium
python3 -m pip install bokeh selenium pandas
# install firefox-geckodriver
sudo apt install firefox-geckodriver

Note: all the commands below are ran from the colcon_ws where performance_test/performance_report is installed.

# Build performance_test and performance_report
colcon build

# Set up the environment
source install/setup.bash

# Run perf_test for each experiment in the yaml file
ros2 run performance_report runner \
  --log-dir perf_logs \
  --test-name experiments \
  --configs src/performance_test/performance_report/cfg/runner/run_one_experiment.yaml

# The runner generates log files to the specified directory: `./perf_logs/experiements/`

# Generate the plots configured in the specified yaml file
ros2 run performance_report plotter \
  --log-dir perf_logs \
  --configs src/performance_test/performance_report/cfg/plotter/plot_one_experiment.yaml

# The generated plots will be saved in `./perf_logs`

# Generate the reports configured in the specified yaml file
ros2 run performance_report reporter \
  --log-dir perf_logs \
  --configs src/performance_test/performance_report/cfg/reporter/report_one_experiment.yaml

The above example only runs and plots one experiement, however multiple experiements can be ran automatically by specifying it in the configuration file.

Try the above commands again, this time replacing the yaml file with *_many_experiements.yaml for each executable.

When running many experiments the runner will by default skip any experiments that already have log files generated in the specified log_dir. This can be overridden by adding the -f or --force argument to the command.

Running the same experiments on multiple platforms

Suppose you want to run an experiment on multiple platforms, then combine the results into a single report. First, pass the --test-name arg to runner, to differentiate the result sets:

# on platform 1:
ros2 run performance_report runner --test_name platform1 -l log_dir -c run.yaml
# results will be stored in ./log_dir/platform1/

# on platform 2:
ros2 run performance_report runner --test_name platform2 -l log_dir -c run.yaml
# results will be stored in ./log_dir/platform2/

You can then combine these results into a single log_dir, on the platform where you will run plotter or reporter. Then, in your plotter or reporter configuration file, set test_name in each dataset, to select results from that platform's result set:

# report.yaml
datasets:
  dataset_p1:
    test_name: platform1  # this matches the --test-name passed to runner
    # other fields...
  dataset_p2:
    test_name: platform2  # this matches the --test-name passed to runner
    # other fields...
reports:
  # ...

ros2 run performance_report reporter -l log_dir -c report.yaml

Notes

  • Currently, this tool is intended for ROS 2 with rmw_cyclone_dds, or Apex.OS with Apex.Middleware. It has not been tested with any other transport.
  • If the run configuration includes SHMEM or ZERO_COPY transport, then a file for configuring the middleware will be created to enable the shared memory transfer.
    • You must start RouDi before running the experiments. This tool will not automatically start it for you.
CHANGELOG

Changelog for package performance_report

X.Y.Z (YYYY/MM/DD)

1.3.5 (2022/12/05)

1.3.4 (2022/11/28)

1.3.3 (2022/11/28)

Fixed

  • Do not try to create a box-and-whisker for a file that contains no measurements

1.3.2 (2022/11/21)

1.3.1 (2022/11/21)

1.3.0 (2022/08/25)

Added

  • The reporter configuration supports box-and-whisker latency plots:
    • set the x_range to Experiment
    • set the y_range to latency_mean
    • set datasets to one or more datasets, each containing a single experiment
    • an example can be found in cfg/reporter/report_many_experiments.yaml ### Changed
  • Expanded the transport setting into the following two settings:
    • process_configuration:
      • INTRA_PROCESS
      • INTER_PROCESS
    • sample_transport:
      • BY_COPY
      • SHARED_MEMORY
      • LOANED_SAMPLES

1.2.1 (2022/06/30)

1.2.0 (2022/06/28)

Changed

  • In the reporter configuration, the template_name value may be an array

1.1.2 (2022/06/08)

1.1.1 (2022/06/07)

Fixed

  • Bokeh line style can be specified in the plotter and reporter .yaml files

1.1.0 (2022/06/02)

Fixed

  • Fix the GBP builds by removing python3-bokeh-pip from package.xml

1.0.0 (2022/05/12)

Added

  • Shared memory experiments are now compatible with both Apex.Middleware and rmw_cyclonedds_cpp
  • commander tool to emit the commands for running the experiments, instead of running them directly ### Changed
  • Use the new perf_test CLI args for QOS settings instead of old flags ### Deprecated ### Removed ### Fixed

Wiki Tutorials

See ROS Wiki Tutorials for more details.

Source Tutorials

Not currently indexed.

Launch files

No launch files found

Messages

No message files found.

Services

No service files found

Plugins

No plugins found.

Recent questions tagged performance_report at answers.ros.org

No version for distro noetic. Known supported distros are highlighted in the buttons above.
No version for distro melodic. Known supported distros are highlighted in the buttons above.
No version for distro ardent. Known supported distros are highlighted in the buttons above.
No version for distro bouncy. Known supported distros are highlighted in the buttons above.
No version for distro crystal. Known supported distros are highlighted in the buttons above.
No version for distro eloquent. Known supported distros are highlighted in the buttons above.
No version for distro dashing. Known supported distros are highlighted in the buttons above.
No version for distro lunar. Known supported distros are highlighted in the buttons above.
No version for distro jade. Known supported distros are highlighted in the buttons above.
No version for distro indigo. Known supported distros are highlighted in the buttons above.
No version for distro hydro. Known supported distros are highlighted in the buttons above.
No version for distro kinetic. Known supported distros are highlighted in the buttons above.